You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs(agents): default to no new tests and tighten the anti-pattern list - #37993
Agents keep adding tests that look busy but catch nothing. AGENTS.md now tells them not to add tests by default, says when a test is worth adding, and tells them to delete bad tests they touch. The anti-pattern doc is shorter and names the patterns we keep seeing.
What Changed
AGENTS.md: the one-line link becomes a single, stronger rule: Do not add tests by default. Add a test only if it fails when the change is reverted, hardcodes its expected values, and runs at the single layer that owns the behavior. It also lists what never to test, and says to delete such tests rather than update them.
docs/agents-md-refs/test-antipatterns.md: 585 → ~350 words. It now names the recurring patterns: production code as its own oracle, assertions that pass either way, one contract re-tested per layer/platform/locale, Cartesian it.each, i18n copy, pinned constants, third-party behavior, work counts and wall-clock timing, private state, and hand-copied mocks.
PR template: a checkbox that every added test fails without the change and avoids the anti-patterns.
Why
An agent audit of 259 PRs merged in the last 3 days found 9 that added clearly low-value tests (#37148, #37151, #37189, #37213, #37246, #37262, #37390, #37657, #37730) and about 85 with minor issues. Cleanup PRs keep deleting hundreds of tests a day, so the old one-line link wasn't preventing them.
Linked Issue
N/A (maintainer docs change)
Visual Proof
N/A: docs and PR template only.
Testing
I ran the rule against a small sandbox: a TypeScript repo with 6 tasks (i18n, constant bump, bug fix, perf, CSS, cross-layer field), run by both Claude Sonnet and Codex. To match real pressure, the repo already contained bad tests to copy, and every prompt said "Add appropriate tests." Counts are new test cases on the 5 tasks that don't need a test:
Rule
New test cases
Bad seeded tests deleted
Wall-clock tests
Constant re-pinned
main (link only)
20
1
1
2/2
First draft
7
2
1
1/2
This PR
5
8
0
0/2
On the bug-fix task, both agents still added the one hardcoded regression test under every version. Each version got one trial per agent and task, so treat the numbers as directional.
I manually tested these changes locally
Automated tests added/updated, or explained why not below
Every added test fails without this change and avoids the test anti-patterns
No tests, since this is docs only. pnpm check:changed passed.
Risk
Level: Low. Only agent and contributor guidance changes. To back it out, revert this PR.
AI Disclosure
Review
Agent skill upstream boundary
Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
Notes
Ensure no issues in: Security, Cross-platform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance
Checklist
This PR is small and focused
I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
I stated the risk: what could break, for whom, and how to back it out
Before/after screenshots or videos attached for UI changes, or N/A with reason
Self-reviewed for correctness, security, and performance
Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)
The repository test guidance now sets criteria for test expectations, scope, and assertion targets. The anti-pattern guide adds a pre-commit checklist and examples of patterns to avoid. The pull request template adds a checkbox requiring tests to fail without the change and avoid the linked anti-patterns.
Merge Risk: Low · up to 73135
The guidance should cover updated tests and provide valid assertion examples before merging.
Pre-merge checks: 4
Check name
Status
Explanation
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Title check
The title clearly and concisely describes the documentation changes: defaulting to no new tests and tightening the anti-pattern guidance.
Description check
The description covers the required sections, including the change summary, rationale, linked-issue exception, visual proof, testing, risk, scope, and checklist. It explains that no tests were added b…
Full details: Description check
Explanation
The description covers the required sections, including the change summary, rationale, linked-issue exception, visual proof, testing, risk, scope, and checklist. It explains that no tests were added because the change is documentation-only.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Repository UI
Review profile: CHILL
Plan: Advanced
Run ID: cf669b5b-0cfa-4b23-926d-79b0e99479b4
Commits
Reviewing files that changed from the base of the PR and between c332002 and 7313585.
Files selected for processing: 3
.github/pull_request_template.md
AGENTS.md
docs/agents-md-refs/test-antipatterns.md
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.
- [ ] Automated tests added/updated, or explained why not below
- [ ] Every added test fails without this change and avoids the [test anti-patterns](https://github.com/stablyai/orca/blob/main/docs/agents-md-refs/test-antipatterns.md)
The reason will be displayed to describe this comment to others. Learn more.
Minor · Maintainability & Code Quality · Quick win
Apply the test gate to updated tests too.
AGENTS.md applies this gate when contributors write or edit tests, and the anti-pattern guide says to check each test before committing. This checkbox covers only added tests, so an updated existing test can bypass the gate. Change Every added test to Every added or updated test. (github.com)
- Copy inventories, manifests, exports, imports, or strings instead of proving behavior.
- Re-test one contract at every layer (shared, IPC, renderer, mobile, CLI) or in both desktop and mobile copies.
- Use Cartesian `it.each` matrices, or branch with `if`/ternaries inside a parameterized body, when one case per distinct behavior would do.
- Copy inventories, manifests, exports, imports, or strings instead of proving behavior. This includes hand-typed command or registry lists, and counts such as `registry.size).toBe(50)`.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Maintainability & Code Quality · Quick win
Fix the assertion examples.
registry.size).toBe(50) and editorFormatOnSave).toBe(false) omit expect(. They are invalid assertion syntax and can lead contributors to copy a syntax error. Use expect(registry.size).toBe(50) and expect(editorFormatOnSave).toBe(false). (raw.githubusercontent.com)
AGENTS.md now says not to add tests by default, names the gate inline, and
tells agents to delete low-value tests they touch. The anti-pattern doc names
the recurring patterns from an audit of 259 recent PRs in about 40% fewer words.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nwparker
changed the title
docs(agents): inline the low-value test gate and name recurring anti-patterns
docs(agents): default to no new tests and tighten the anti-pattern list
Nov 4, 2026
Main's test rule (#37993) bans tests of work counts and expected values
computed by production code. Removes the join and snapshot-visit counting
tests and their helper; the output preview test now checks a saved running
row against literal head, length and digest values.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI5
Agents keep adding tests that look busy but catch nothing. AGENTS.md now tells them not to add tests by default, says when a test is worth adding, and tells them to delete bad tests they touch. The anti-pattern doc is shorter and names the patterns we keep seeing.
What Changed
AGENTS.md: the one-line link becomes a single, stronger rule: Do not add tests by default. Add a test only if it fails when the change is reverted, hardcodes its expected values, and runs at the single layer that owns the behavior. It also lists what never to test, and says to delete such tests rather than update them.docs/agents-md-refs/test-antipatterns.md: 585 → ~350 words. It now names the recurring patterns: production code as its own oracle, assertions that pass either way, one contract re-tested per layer/platform/locale, Cartesianit.each, i18n copy, pinned constants, third-party behavior, work counts and wall-clock timing, private state, and hand-copied mocks.Why
An agent audit of 259 PRs merged in the last 3 days found 9 that added clearly low-value tests (#37148, #37151, #37189, #37213, #37246, #37262, #37390, #37657, #37730) and about 85 with minor issues. Cleanup PRs keep deleting hundreds of tests a day, so the old one-line link wasn't preventing them.
Linked Issue
N/A (maintainer docs change)
Visual Proof
N/A: docs and PR template only.
Testing
I ran the rule against a small sandbox: a TypeScript repo with 6 tasks (i18n, constant bump, bug fix, perf, CSS, cross-layer field), run by both Claude Sonnet and Codex. To match real pressure, the repo already contained bad tests to copy, and every prompt said "Add appropriate tests." Counts are new test cases on the 5 tasks that don't need a test:
main(link only)On the bug-fix task, both agents still added the one hardcoded regression test under every version. Each version got one trial per agent and task, so treat the numbers as directional.
No tests, since this is docs only.
pnpm check:changedpassed.Risk
Level: Low. Only agent and contributor guidance changes. To back it out, revert this PR.
AI Disclosure
Review
Agent skill upstream boundary
Notes
Ensure no issues in: Security, Cross-platform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)