Org standard for automatically handling spam / low-effort outside PRs

[don-petry]

Summary

Public org repos accept PRs from anyone. Spam and low-effort PRs currently sit open until a
maintainer notices them, attract third-party bot noise, and can consume automation budget. This
proposes an org standard: deterministic scoring, staged actions (label, then close), and a
dry-run rollout, so maintainers only see PRs worth their time.

Motivating incident

petry-projects/.github-private#1976 (2026-09-29):

• Title was a person's name ("Yasir Bin Mudassir"); body was the unfilled PR template.
• The only change was one README word altered to a typo ("Invoke with" became "rewoke with").
• Opened from the fork's main by a brand-new account (FIRST_TIME_CONTRIBUTOR); created and
  updated within about 90 seconds.
• Nothing in our automation acted on it. Only third-party bots (Qodo, CodeRabbit, cubic) reacted.
  stale-manager would not have closed it for about 37 days.

Findings from the existing tooling (.github-private)

1. The description-quality check is fooled by an unfilled template. sc_description_missing
   (scripts/lib/safety-checks.sh) greps the body for words like problem, risk and
   rollback. The template's own headings and HTML comments contain them, so an empty template
   reports 0 of 5 sections missing.
2. No workflow labels, closes or reports low-quality outside PRs.
3. Fork PRs do not reach our agents (no secrets), so the risk is noise and maintainer time,
   not agent cost, unless a maintainer labels or dispatches something on them.

Proposed standard

Layer 0: GitHub-native settings (no code)

• Require approval for workflows from all outside contributors.
• Restrict PR creation to prior contributors where the plan allows it.
• Maintainers block and report confirmed repeat abusers manually. Automation never blocks users.

Layer 1: spam-pr-guard workflow (Class 1, event-driven)

• Trigger: pull_request_target on opened, edited, reopened. No schedule.
• Scope: only authors with author_association of NONE, FIRST_TIME_CONTRIBUTOR or
  FIRST_TIMER, or PRs from forks. Never members, collaborators or bots.
• Security: no checkout and no PR code execution; API and metadata only; permissions limited to
  pull-requests: write and issues: write.
• Scoring (scripts/spam-pr-score.sh, deterministic, no LLM):

Signal	Points
Template body with every section empty (comments and headings stripped)	+3
Title is a bare name / single word / unrelated to the diff	+2
Diff of 3 lines or fewer in README/docs only, no linked issue	+2
Meaningless or typo-introducing edit	+1
Head branch is the fork's default branch	+1
Account under 30 days old with no other activity	+1
Burst: more than N open PRs from the author across the org in 24h	+3
Touches workflows, secrets or install scripts	Escalate to human; never auto-close

• Actions: low = nothing; medium = needs-info label plus one marker-keyed comment asking
  for the missing sections; high = spam label, explanatory comment (with how to reopen),
  close as not_planned.
• Ships in log-only DRY_RUN mode for about 2 weeks before label/close is enabled.

Layer 2: fix the description check

Strip HTML comments and headings before matching, and require real text under each section.
Add a bats test using the #1976 body as a fixture.

Layer 3: backstop and reporting

• Short stale window (about 3 days) for outside PRs labelled needs-info with no response.
• Weekly count of auto-closed PRs and any reopened false positives, to tune thresholds.

Design constraints

• The new workflow is an agentic role: it needs a §4 row in docs/agentic-interaction-model.md
  and an interaction-contracts/<name>.yml.
• Fail closed: if a signal cannot be read, escalate to a human rather than treat it as "not spam".
• Verify by execution: tests must include a counterfactual (a legitimate first-time PR must score
  low).

Rollout

1. Layer 0 settings; close #1976 by hand.
2. PR: description-check fix and tests.
3. PR: scoring script and tests (fixtures for #1976 and a legitimate PR).
4. PR: workflow in dry-run, plus §4 row and interaction contract.
5. After about 2 weeks: review the log, tune thresholds, enable label/close.
6. PR: stale-manager tweak and reporting.

Open questions

1. Should high-score PRs be closed automatically, or only labelled for a human to close?
2. Should a maintainer be notified for each auto-close?
3. Scope: .github-private only, or every public org repo (via a reusable workflow and a
   caller stub)?
4. Where should the standard live (standards/ in this repo)?
5. Threshold values and the burst limit N.

[github-actions[bot]]

Enhancement: Market Context, Prior Art, and Implementation Options

Problem Framing — Sharper

The spam/low-effort PR problem has escalated from a seasonal Hacktoberfest nuisance to a year-round structural challenge. GitHub reported site-wide merged PRs grew from 25 million/month (January 2023) to 90 million/month (March 2026) — a 3.6× increase driven partly by AI-assisted contributions. High-profile projects have already taken drastic measures: curl shut down its bug bounty after AI hallucinations comprised 20% of security reports, Apache Airflow saw fake issues nearly double from AI submissions, and Ghostty implemented zero-tolerance for drive-by AI PRs with permanent bans for repeat offenders.

The proposal's motivating incident (a name-as-title, template-unfilled, typo-introducing PR from a brand-new account) is textbook — and the 37-day stale-manager gap means noise compounds. This is the right time to build a layered defense.

Prior Art & Competitive Landscape

GitHub-native controls (shipped since the proposal was drafted):

Feature	Status	What it does	Gap it leaves
PR caps (GitHub Blog, June 2026)	GA	Per-repo ceiling on concurrent open PRs from users without write access. Bypass list for trusted contributors.	Caps quantity, not quality — a single low-effort PR still gets through. No org-wide setting yet.
PR archiving (July 2026)	GA	Hide closed/locked spam PRs from default views (404 for non-admins). Preserves data for compliance.	Manual action — no automated trigger.
Restrict PRs to collaborators (Changelog, Feb 2026)	GA	Blocks all outside PRs entirely.	Too blunt for repos that welcome genuine first-time contributors.
Issue limits	In development	Similar caps for issue creation.	Not yet shipped.
Smarter bypass signals	Exploring	Auto-exempt based on account age, prior merges, org membership.	Not yet shipped. Cross-repo controls (spray-pattern detection) also in exploration.
Low Quality comment classification	GA	New moderation option distinct from spam/abuse for comments.	Covers comments, not PRs.
actions/checkout v7 safer defaults (Changelog, June 2026)	GA (enforcement July 20)	Blocks fork PR checkout in pull_request_target by default. Requires allow-unsafe-pr-checkout flag.	This is a security win that validates the proposal's metadata-only pull_request_target design.

Community tooling:

Tool	Approach	Strengths	Weaknesses
anti-slop	34 heuristic rules across PR structure, content quality, contributor signals, merge ratios. Configurable failure threshold (default: 4 failures).	Most comprehensive deterministic scorer. Checks account age, profile completeness, daily fork rate, merge ratio, commit density.	Opinionated (Conventional Commits enforced); some rules (emoji count, max description length) may over-fire on legitimate PRs.
Spam Check Action	Hacktoberfest-era heuristics — fork + new account + only forked repos.	Simple, focused.	Narrow scope — misses AI slop, empty-template PRs.
Spamtoberfest	Labels spam PRs based on fork + registration date + profile heuristics.	Good false-positive calibration for seasonal spam.	Seasonal focus; not designed for year-round operation.
AntiSpam PR Labeler	Labels + comments on detected spam.	Marker-based idempotency.	Limited signal set.

Key takeaway: GitHub's native PR caps handle the volume axis well but leave a quality gap — exactly what the proposed spam-pr-guard scoring fills. The proposal is complementary to, not duplicated by, the platform roadmap.

Options Table

Option	Description	Pros	Cons
A. Native-only	Enable PR caps + collaborators-only on sensitive repos. No custom workflow.	Zero maintenance. Uses GitHub's own signals.	No quality scoring. Blocks legitimate first-timers on collaborators-only repos. PR caps don't exist at org level yet.
B. Proposed spam-pr-guard (this idea)	Custom deterministic scoring workflow with staged label→close actions.	Granular, tunable, org-specific signals. DRY_RUN rollout. Works today.	Must be maintained. False-positive risk on legitimate first-timers. Another workflow to audit.
C. Adopt anti-slop as-is	Use the community action directly.	34 rules, battle-tested on real repos. Fast to deploy.	Opinionated defaults (Conventional Commits, emoji limits). Less control over org-specific scoring weights. External dependency.
D. Hybrid: native caps + lightweight custom scorer	Enable PR caps for volume defense; build a slim custom scorer (5–8 signals max) for quality defense. Lean on GitHub's upcoming smarter bypass signals when they ship.	Best coverage. Smallest custom surface. Future-proof — can retire custom scorer as GitHub ships more.	Still requires some custom code. Two systems to reason about.

Recommendation: Option D offers the best risk/reward. It layers GitHub's volume controls with a focused custom quality gate, and the custom component is designed to shrink over time as GitHub ships smarter bypass signals and cross-repo controls. The proposal's scoring table is a strong starting point — just trim it to the highest-signal checks.

Risks & Unknowns

1. False positives on legitimate first-timers. This is the biggest risk. The anti-slop project's experience shows that profile completeness scoring and merge-ratio checks can misfire on students, developers from regions where GitHub profiles are sparse, and first-time open source contributors. Mitigation: The proposal's staged approach (label → comment with "how to reopen" → close) is correct. Add a 24-hour grace period before auto-close, and ensure the comment includes a clear appeal path (mention a maintainer or remove the label). Track the false-positive rate in the weekly reporting (Layer 3).

2. Template-body bypass. The proposal correctly identifies that the existing sc_description_missing check is fooled by unfilled templates. However, stripping HTML comments and headings (Layer 2) is necessary but not sufficient — AI-generated PRs can fill template sections with plausible but empty-calorie text. Consider also checking whether the description references the actual diff (e.g., mentions changed filenames or line numbers).

3. Security-sensitive file escalation scope. The scoring table flags PRs that "touch workflows, secrets or install scripts" for human escalation. This is good, but the definition of "security-sensitive" should be explicit and maintained in a config file (aligned with the existing workflow-exemptions.json pattern) rather than hardcoded in the scoring script.

4. pull_request_target security surface. The proposal's "no checkout and no PR code execution; API and metadata only" constraint is exactly right, and is now validated by GitHub's own actions/checkout v7 defaults (which block fork PR checkout in pull_request_target by default). This design choice should be called out as a security invariant in the interaction contract, not just a design note.

5. Org-wide vs. per-repo scope (Open Question Add coding standards & principles section to AGENTS.md #3). The strongest approach: build as a Tier 1 reusable workflow (matching the existing standards/workflows/ pattern), with a thin caller stub per repo. This gives central control over scoring logic while letting each repo opt in. The pr-limits standard already demonstrates this pattern — the gate library lives centrally, consumers read config at runtime.

Phased Validation Path

Phase	Duration	Gate to next
0. Enable native PR caps	1 day	Caps set on all public repos. Bypass list includes dependabot[bot] and org-leads.
1. Description-check fix	1 sprint	Bats tests pass with #1976 body as fixture AND a legitimate first-time PR scores low.
2. Scoring script + DRY_RUN workflow	1 sprint	Script handles all fixture cases. Workflow runs on pull_request_target with permissions: pull-requests: write. §4 row and interaction contract merged.
3. Observe & tune	2–3 weeks	Review DRY_RUN log. Zero false positives on legitimate PRs. Thresholds validated against ≥10 real events.
4. Enable label + close	1 day	Flip DRY_RUN off. needs-info at medium, spam + close at high.
5. Backstop + reporting	1 sprint	3-day stale window for needs-info with no response. Weekly report metric wired to step summary.
6. Reusable workflow promotion	1 sprint	Reusable workflow + caller stub in standards/workflows/. Downstream repos opt in.

Concrete Success Metrics

Metric	Baseline (current)	Target (3 months post-enable)
Median time-to-close for spam PRs	~37 days (stale-manager)	< 24 hours
Maintainer-initiated spam closes per month	Manual, uncounted	0 (automated)
False positive rate (legitimate PRs auto-closed)	N/A	< 2% of auto-closes
Third-party bot noise on spam PRs	Unbounded (CodeRabbit, Qodo react to all)	Near zero (closed before bots trigger)
Reopened-after-auto-close rate	N/A	Tracked; target < 5%

Answers to Open Questions

1. Auto-close vs. label-only for high-score PRs? Auto-close with a clear comment explaining why and how to reopen. The comment is the safety net. Label-only still requires maintainer intervention, which defeats the purpose. But add a 24-hour delay between labeling and closing to catch false positives.

2. Maintainer notification per auto-close? No individual notifications — this would recreate the noise problem. Instead, aggregate in the weekly report (Layer 3). If a PR is reopened after auto-close, then notify a maintainer (this is the signal that the scorer may have misfired).

3. Scope: .github-private only or all public repos? All public repos, via a Tier 1 reusable workflow + caller stub (matching the existing centralization pattern). But roll out via canary rings: .github-private first (ring 0), then other repos after the 2-week soak.

4. Where should the standard live? standards/spam-pr-guard.md in this repo, with a companion standards/spam-pr-guard.json for the scoring weights and threshold config (following the pr-limits.md + pr-limits.json pattern).

5. Threshold values and burst limit N? Start conservative: medium ≥ 4 points, high ≥ 6 points, burst N = 3 PRs in 24h. Tune during the DRY_RUN phase based on observed distribution. The config-file approach means threshold changes are a JSON edit, not a code change.

[don-petry]

@petry-projects/solution-architect - Review and add your perspective & goals

[don-petry]

📋 Initiative planned by the BMAD Scrum Master (Bob).

Epic #1200 — Org standard: automated spam / low-effort outside-PR handling (spam-pr-guard)

7 stories created (inert — labelled initiative, NOT initiative:auto):

• [Phase 0] Layer 0 GitHub-native controls and close the motivating incident #1201 (S) — [Phase 0] Layer 0 GitHub-native controls and close the motivating incident
• [Phase 1] Fix the description-quality check so an unfilled template is not read as complete #1202 (M) — [Phase 1] Fix the description-quality check so an unfilled template is not read as complete
• [Phase 2] Deterministic spam-PR scoring script and config #1203 (L) — [Phase 2] Deterministic spam-PR scoring script and config
• [Phase 3] spam-pr-guard workflow in DRY_RUN, with the section 4 row and interaction contract #1204 (M) — [Phase 3] spam-pr-guard workflow in DRY_RUN, with the section 4 row and interaction contract
• [Phase 4] Repo-local standard document for spam-pr-guard #1205 (S) — [Phase 4] Repo-local standard document for spam-pr-guard
• [Phase 5] DRY_RUN soak, threshold tuning without overfitting, and go/no-go to enable label/close #1206 (M) — [Phase 5] DRY_RUN soak, threshold tuning without overfitting, and go/no-go to enable label/close
• [Phase 6] Stale backstop for needs-info and weekly reporting #1207 (M) — [Phase 6] Stale backstop for needs-info and weekly reporting

Open questions for review:

• Scope: org-wide vs .github-private-only. This plan implements ring-0 (.github-private) first and defers promotion to a Tier-1 reusable workflow + caller stub (enhancement Option D). Confirm the org-wide rollout rings and whether promotion is in scope for this epic or a follow-up.
• Final home of the standard: repo-local docs vs standards/spam-pr-guard.md + standards/spam-pr-guard.json in petry-projects/.github (discussion open Q4). The plan uses repo-local-first; confirm the promotion target.
• Auto-close vs label-only for high-score PRs, and any close delay (discussion open Q1). Handled as a recorded go/no-go in Phase 5 rather than baked into code; confirm the intended default (enhancement recommends auto-close with a reopen comment plus a delay).
• Maintainer notification policy (discussion open Q2): the plan surfaces aggregate weekly reporting plus a reopen-after-close signal and no per-close notification. Confirm this is the desired policy.

---

Review the epic and its sub-issue DAG, adjust as needed, then add initiative:auto to epic #1200 to hand it to initiative-driver for auto-implementation.