Skip to content

docs(ams): inventory real-world contribution-eligibility signals across a repo sample #6877

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bitfathers94 wants to merge 1 commit into from
Closed

docs(ams): inventory real-world contribution-eligibility signals across a repo sample #6877

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 215 additions & 0 deletions packages/loopover-miner/docs/ams-contribution-signal-inventory.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
# AMS contribution-profile: real-world eligibility-signal inventory

Research inventory for **#6794**. Evidence-gathering only — the `ContributionProfile` schema this informs is
designed in the dependent issue (#6795), and no code changes are part of this deliverable.

## Purpose

Before AMS can build a per-repo contribution profile (the #6793 epic), we need an accurate, evidence-based
picture of what real repositories actually expose as PR-eligibility signals — which signal types exist, where
they live, and how consistently they carry machine-extractable meaning versus prose a human has to interpret.
This is an audit of real examples, not an assumption of what "most repos" do.

## Sample

Ten public repositories, chosen for a spread of contribution norms, sizes, and documentation maturity:

| Repo | Why in the sample |
| -------------------------- | ------------------------------------------------------------------------ |
| `JSONbored/loopover` | Gate-enabled, this project's own convention (the reference case) |
| `JSONbored/metagraphed` | Gate-enabled sibling in the same gate family |
| `facebook/react` | Large OSS, mature contribution process, heavy AI-agent tooling |
| `rust-lang/rust` | Very large, deeply structured label taxonomy (~960 labels) |
| `microsoft/vscode` | Very large label taxonomy (~700 labels), triage-automation heavy |
| `cli/cli` | Mid-size OSS with an explicit, prose-stated contributor-eligibility rule |
| `denoland/deno` | Mid-size OSS, triage-state-oriented labels, no first-issue vocabulary |
| `honojs/hono` | Small OSS, minimal label set, generic docs |
| `tailwindlabs/tailwindcss` | Small OSS with a "discuss first, PRs likely closed" norm |
| `sindresorhus/ky` | Minimal library — no repo-local contribution docs at all |

`ljharb/qs` was also spot-checked as a second no-local-docs case and behaves like `ky`.

## Method

Per repo, via the GitHub REST API and the git-trees API (authoritative file list, not per-path `contents`
probes, which false-positive on case-insensitive/redirected paths):

- **Labels** — `GET /repos/{owner}/{repo}/labels` (paginated), reading each label's `name` **and**
`description`, judging eligibility/scope meaning from those two fields alone.
- **Docs presence** — `GET /repos/{owner}/{repo}/git/trees/{default_branch}?recursive=1`, filtered for
`CONTRIBUTING.md`, `.github/PULL_REQUEST_TEMPLATE(.md|/)`, `.github/ISSUE_TEMPLATE/`, `AGENTS.md`,
`CLAUDE.md`, `.claude/**`, `.cursorrules`/`.cursor/rules`, `CODEOWNERS`.
- **Docs content** — fetched `CONTRIBUTING.md` and the PR template for each repo that has them, and read for
the four eligibility questions below.

## The four eligibility questions (per repo)

1. **Label taxonomy** — do any labels read as eligibility/scope signals from name + description alone?
2. **`CONTRIBUTING.md`** — does it state a linked-issue requirement, a required-label rule, or assignment/claim
rules?
3. **PR template** — is there one, and does it carry an eligibility checkbox (e.g. "link the issue this closes")?
4. **AI-agent-facing docs** — `AGENTS.md` / `CLAUDE.md` / `.claude/skills/**` / `.cursor/rules` that state
contribution rules explicitly for an AI contributor (as opposed to general build/dev guidance)?

## Master inventory

`Y` = present and eligibility-bearing · `dev` = present but developer/build guidance only (no
contributor-eligibility rules) · `soft` = present but optional/advisory · `—` = absent.

| Repo | Eligibility labels | Exclusion labels | CONTRIBUTING eligibility rule | PR template | Linked-issue requirement | Agent-facing contribution docs |
| -------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------- | --------------------------------------------------------- | ------------------ | --------------------------------------------- | ------------------------------------------------ |
| `JSONbored/loopover` | **Y** (machine-readable in description) | **Y** (`maintainer-only`) | **Y** (linked open eligible issue required) | Y (checkbox) | **Required (hard)** | **Y** (`CLAUDE.md`/`AGENTS.md`/`.claude/skills`) |
| `JSONbored/metagraphed` | **Y** (machine-readable in description) | **Y** (`maintainer-only`) | **Y** (linked open eligible issue required) | Y (multi-template) | **Required (hard)** | **Y** (`CLAUDE.md`/`AGENTS.md`/`.claude/skills`) |
| `cli/cli` | Y (`help wanted`, `good first issue`) | **Y** (`core` = "not accepting PRs from outside contributors") | **Y** ("PRs accepted for issues labelled `help wanted`") | Y | soft (`Fixes #N`) | dev (`AGENTS.md`) |
| `microsoft/vscode` | Y (`help wanted`, `good first issue`) | **Y** (`team-low-hanging` = "No external contributions will be accepted") | — | Y | soft ("Associate an issue") | dev (`AGENTS.md`) |
| `rust-lang/rust` | Y (rich `E-*` family) | — (uses `S-*`/`needs-*` workflow states, not a "no PRs" label) | — (redirects to dev-guide) | Y | **soft (explicitly optional)** | dev (only in `rust-analyzer` subtree) |
| `denoland/deno` | soft (triage-state only; no first-issue vocab) | — | — (anti-AI-spam clause only) | Y | soft ("ensure a related issue is referenced") | dev (`CLAUDE.md`/`.claude/skills`) |
| `facebook/react` | Y (`good first issue`, `Difficulty:*`) | — | — (generic) | Y | — | dev (extensive `.claude/skills`, build tasks) |
| `honojs/hono` | soft (`help wanted`, empty description) | — | — (generic) | Y | — | — |
| `tailwindlabs/tailwindcss` | soft (`good first issue`, empty description) | — | **negative** ("discuss first; feature PRs likely closed") | Y | — | — |
| `sindresorhus/ky` | soft (GitHub-default `good first issue`/`help wanted`) | — | — (none repo-local) | — | — | — |

## Findings by signal type

### 1. Labels — the name is portable; the description usually is not

- **Eligibility label _names_ are the most common cross-repo signal.** `good first issue` and/or `help wanted`
appear in 7 of 10 repos (react, vscode, cli, hono, tailwind, ky, and loopover carries `help wanted` too).
They are the closest thing to a universal, name-only eligibility hint.
- **Label _descriptions_ are unreliable as an eligibility source in almost every repo.** They are frequently
empty (react's `Difficulty:*` and `good first issue`, hono's `help wanted`, tailwind's `good first issue`),
or they restate GitHub's stock defaults with no repo-specific meaning (`ky`: `good first issue` = "Good for
newcomers", `help wanted` = "Extra attention is needed"). Where descriptions do carry content, the wording
is inconsistent even for the same label name — `help wanted` is "Contributions welcome" (cli), "good
community contribution opportunities" (vscode), or empty (hono).
- **Only the two gate-enabled repos put machine-readable eligibility semantics _in the description_.**
`loopover` and `metagraphed` are alone in the sample in encoding, directly in each label's description text,
whether an issue is contributor-eligible, what scope it belongs to, and its relative priority — plus an
explicit `maintainer-only` label whose description states the work is not contributor-eligible. This is the
only place in the whole sample where a label description alone is sufficient to decide eligibility
programmatically.
- **Large taxonomies dilute rather than clarify.** rust (~960 labels) and vscode (~700) have rich vocabularies,
but eligibility signal is spread thin: rust's `E-*` "Call for participation" family (`E-easy`, `E-mentor`,
`E-help-wanted`, `E-needs-*`) is unusually well-described, yet most of the taxonomy is triage/team/status
state irrelevant to a contributor. A profile builder must _select_ the eligibility-bearing labels out of a
large set, not treat label count as signal strength.
- **Exclusion is expressed more consistently than inclusion, when it is expressed at all.** Three repos state a
hard "not for outside contributors" signal, and all three put it somewhere machine-readable: `loopover`/
`metagraphed`'s `maintainer-only` label, `cli/cli`'s `core` label ("This issue is not accepting PRs from
outside contributors"), and `vscode`'s `team-low-hanging` ("No external contributions will be accepted"). The
last two are notable: a non-loopover repo _can_ carry an exclusion rule in a label description.

### 2. `CONTRIBUTING.md` — present often, eligibility-bearing rarely

- 8 of 10 repos have a `CONTRIBUTING.md` (only `ky`/`qs` have none repo-local). Presence is common; a
contributor-eligibility _rule_ inside it is not.
- **Explicit label-gated eligibility in prose exists but is rare.** `cli/cli` is the clearest non-loopover
example: "We accept pull requests for issues labelled `help wanted`" — a directly extractable rule, and it
agrees with the `core` label description. `loopover`/`metagraphed` state the linked-open-eligible-issue
requirement in prose _and_ enforce it at the gate.
- **Negative eligibility rules occur.** `tailwindcss` tells contributors to open a discussion first and warns
that unsolicited feature PRs are "likely to close" — a real eligibility signal, but stated only as prose
discouragement with no label or template mechanism behind it.
- **Most CONTRIBUTING files are generic.** react and hono cover bug-reporting and setup but state no
linked-issue or label gate. rust's top-level `CONTRIBUTING.md` largely redirects to the rustc-dev-guide,
where the real norms (E-labels, `@rustbot claim`) live — i.e. the eligibility norm can sit one hop away from
the file a scraper would first read.

### 3. PR templates — near-universal, but a weak eligibility source

- 9 of 10 repos have a PR template (all but `ky`). So "has a PR template" is almost no signal at all.
- **Linked-issue enforcement strength is a spectrum, and the template wording reveals it:**
- **Hard-required:** loopover/metagraphed — an explicit checkbox ("I linked a currently open issue this PR
resolves … required for every contributor PR"), backed by the gate.
- **Requested:** deno ("Ensure there is a related issue and it is referenced"), vscode ("Associate an issue
with the Pull Request"), cli (`Fixes #NUMBER`).
- **Explicitly optional:** rust ("If you don't know of a related tracking issue … feel free to ignore this").
- **Absent:** react, hono, tailwind ask only for a summary/validation, with no issue field.
- Only loopover/metagraphed carry an eligibility _checklist_ (scope, linked-issue, secret-hygiene boxes). The
rest are description prompts, not eligibility gates.

### 4. AI-agent-facing docs — increasingly common, almost never about eligibility

- **Agent-facing files are now common:** 7 of 10 repos carry at least one of `AGENTS.md`, `CLAUDE.md`, or a
`.claude/skills/**` tree (loopover, metagraphed, react, vscode, cli, deno, and rust in its `rust-analyzer`
subtree). hono, tailwind, and ky have none.
- **But their _purpose_ splits sharply.** In 5 of those 7, the agent docs are developer/build guidance — how to
run the compiler, extract error codes, format, triage, port a pass (react's `.claude/skills/{flow,
feature-flags,extract-errors}` and `compiler/.claude/**`; deno's `.claude/skills/{fmt,lint,node-compat,
review-pr}`; vscode/cli `AGENTS.md`). None of these tell an external AI contributor which _issues_ are
eligible to work on or what a PR must satisfy to be accepted.
- **Only loopover/metagraphed use agent docs to state contribution _eligibility_ rules for an AI contributor.**
Their `CLAUDE.md` + `AGENTS.md` + `.claude/skills/contributing-to-*` spell out the linked-issue requirement,
the scored-label convention, scope boundaries, and the one-shot gate — exactly the rules a profile would need.
This is the richest agent-facing eligibility source in the sample, and it is unique to this gate family.

## Consistency analysis

- **Signal _location_ is inconsistent.** The same fact ("which issues may I PR against?") lives in a label
description in one repo, in `CONTRIBUTING.md` prose in another, in a dev-guide one hop from CONTRIBUTING in a
third, and nowhere explicit in a fourth. No single file is a reliable sole source.
- **Signal _shape_ is inconsistent.** Label descriptions range from machine-readable eligibility+scope+priority
(loopover) to empty (hono/tailwind) to stock GitHub defaults (ky). A profile builder cannot assume a
description means anything.
- **Name-level conventions are the most portable thing.** `good first issue` / `help wanted` as label _names_,
and a `Closes/Fixes #N` phrase in a PR body, are the two conventions that recur across unrelated repos — but
both are weak: the label names carry no enforcement, and the closing keyword is optional in most repos.
- **Explicit, enforceable, machine-readable eligibility is the exception, not the norm** — in this sample it is
essentially unique to the gate-enabled repos, precisely the case AMS already handles. Every other repo sits
somewhere on a gradient from "prose hints a human must interpret" (cli, tailwind, rust) down to "no explicit
signal at all" (ky/qs), which is the case the epic's "deliberately conservative when no profile can be
inferred" requirement exists for.

## Implications for the `ContributionProfile` schema (#6795)

Grounded in the above, the next issue's schema should assume:

1. **Multi-source extraction is mandatory**, not optional — no single file suffices. The profile must be able to
draw eligibility rules from labels, `CONTRIBUTING.md`, the PR template, and agent-facing docs, and merge
them.
2. **Per-rule source provenance matters**, because the _same_ rule can come from a strong source (an enforced
label) or a weak one (an optional prose sentence). The schema already lists provenance in #6795's draft; the
evidence confirms it is essential for the confidence calculation, not just debuggability.
3. **Confidence must be first-class and usually low.** Only 2 of 10 repos yield a high-confidence, fully
machine-readable profile. The common case is a partial/inferred profile (a `help wanted` name with no
enforcement) or none at all — so `discover` must treat an uncertain profile conservatively by design.
4. **Model inclusion and exclusion rules separately.** Exclusion signals (`maintainer-only`, `core`,
`team-low-hanging`) are expressed more consistently and machine-readably than inclusion signals, and a false
negative on an exclusion rule (PRing a `core`/`maintainer-only` issue) is exactly the auto-close waste the
epic exists to prevent — so exclusion should be extracted and weighted at least as strongly as inclusion.
5. **Label _descriptions_ deserve extraction but not trust by default.** Parse them (loopover-shaped repos put
real semantics there), but gate any eligibility inference drawn from a description on corroboration or a low
confidence score, since most repos' descriptions are empty or generic.
6. **Name-level heuristics (`good first issue` / `help wanted`) are a reasonable low-confidence default** for
the majority of repos that expose nothing stronger, but must never be treated as an enforcement guarantee.

## Appendix — raw per-repo notes

- **loopover / metagraphed:** ~27 labels; `gittensor:*` label descriptions carry eligibility+scope+relative-
priority semantics and `maintainer-only` marks non-contributor work; `CONTRIBUTING.md` + PR-template checkbox
require a linked open eligible issue; full `CLAUDE.md`/`AGENTS.md`/`.claude/skills/contributing-to-*` for AI
contributors. metagraphed additionally uses a `.github/PULL_REQUEST_TEMPLATE/` directory (backend/docs/
provider/surface variants).
- **facebook/react:** 76 labels (`good first issue`, `Difficulty: starter|medium|challenging`, all empty
descriptions); generic `CONTRIBUTING.md`; PR template asks only how the change was verified; large
`.claude/skills/**` + `compiler/.claude/{rules,agents,skills}` — all build/dev tasks.
- **rust-lang/rust:** ~960 labels; well-described `E-*` "Call for participation" family is the eligibility
vocabulary; top-level `CONTRIBUTING.md` redirects to the dev-guide; PR template's tracking-issue line is
explicitly optional; agent docs only under `src/tools/rust-analyzer/`.
- **microsoft/vscode:** ~700 labels; `good first issue`/`help wanted` described for contributors, and
`team-low-hanging` explicitly excludes external PRs; no root `CONTRIBUTING.md` eligibility rule; PR template
says "Associate an issue"; `AGENTS.md` is dev guidance.
- **cli/cli:** 80 labels; `help wanted` = "Contributions welcome", `core` = "not accepting PRs from outside
contributors", `help wanted candidate` = not-yet-ready; `CONTRIBUTING.md` states PRs are accepted for
`help wanted` issues; PR template uses `Fixes #NUMBER`; `AGENTS.md` is dev guidance.
- **denoland/deno:** 121 labels, triage-state oriented (`needs triage`, `needs discussion`, `feat` =
accepted), no first-issue vocabulary; `CONTRIBUTING.md` has an anti-AI-spam clause; PR template asks for a
referenced related issue; `CLAUDE.md` + `.claude/skills/**` are maintenance tasks.
- **honojs/hono:** 24 labels (`help wanted`/`triage`, empty descriptions); generic `docs/CONTRIBUTING.md`; PR
template is a tests/format checklist; no agent docs.
- **tailwindlabs/tailwindcss:** 17 labels (`good first issue` empty); `CONTRIBUTING.md` directs feature ideas
to Discussions and warns unsolicited feature PRs are likely closed; PR template is a summary prompt; no agent
docs.
- **sindresorhus/ky:** 13 labels, all GitHub/Issuehunt defaults; no repo-local `CONTRIBUTING.md`, PR template,
issue template, or agent docs (org-level `.github` defaults may apply, but nothing is exposed at the repo
level). `ljharb/qs` behaves the same way.