Skip to content

[Bug]: penalty label multipliers (<1) are floored to 1 and never applied #994

Description

@galuis116

[Bug]: penalty label multipliers (< 1) are silently floored to 1 and never applied — over-rewards low-value PRs

Summary

selectLabelMultiplier computes the per-PR label multiplier as
Math.max(fallback || 1, ...matchedMultipliers). Because the neutral fallback is
always >= 1, any configured penalty label multiplier below 1 (e.g.
refactor: 0.5, docs: 0.3) is clamped back up to 1 and never applied. Only
bonus multipliers (>= 1) ever take effect. A maintainer who configures
penalty labels to dampen low-value PRs gets them silently ignored, and the
contributor's token score / merged-PR estimate is inflated by the dropped
penalty.

The product clearly intends sub-1 multipliers to work: the registry schema allows
any number, the upstream registry sync parses label_multipliers verbatim, the
ruleset treats labelMultipliers as first-class scoreability config — and the
codebase's own scoring fixture ships refactor: 0.5.

Evidence

// src/scoring/preview.ts:736
function selectLabelMultiplier(labels: string[], multipliers: Record<string, number>, fallback: number): number {
  const normalized = new Set(labels.map((label) => label.toLowerCase()));
  return Math.max(
    fallback || 1,                                                              // floor is always >= 1
    ...Object.entries(multipliers).flatMap(([label, multiplier]) => (normalized.has(label.toLowerCase()) ? [multiplier] : [])),
  );
}

The result multiplies the base score:

// src/scoring/preview.ts:320 + :353
const labelMultiplier = selectLabelMultiplier(input.labels ?? [], config?.labelMultipliers ?? {}, config?.defaultLabelMultiplier ?? 1);
...
baseScore * labelMultiplier * issueMultiplier * credibilityMultiplier * reviewPenaltyMultiplier * openPrMultiplier * openIssueMultiplier * timeDecayMultiplier

Verified by executing the exact logic against the config { bug: 1.2, refactor: 0.5, docs: 0.3 }, fallback 1:

Labels Intended multiplier Actual
["refactor"] 0.5 1
["docs"] 0.3 1
["bug", "refactor"] (penalty present) 1.2 (0.5 dropped)
["bug"] 1.2 1.2 ✓
["feature"] (no match) 1 1 ✓

So a refactor PR that should score baseScore * 0.5 instead scores
baseScore * 12× the intended reward; a docs PR at 0.3 scores ~3.3×.

The schema imposes no lower bound (so the config is valid):

// src/openapi/schemas.ts:40
labelMultipliers: z.record(z.string(), z.number()),

Reachability

Fully reachable from the public scoring surface. labelMultipliers is synced from
the upstream registry per repo (src/registry/normalize.ts, src/upstream/ruleset.ts:573),
and input.labels is caller-supplied to buildScorePreview via the MCP
score_preview tool and the score-preview/explain-breakdown routes. Any repo whose
registry config sets a penalty label multiplier gets it silently dropped on every
score preview/breakdown for a PR carrying that label.

Suggested fix

Apply the matched multiplier(s) when any label matches, and fall back to the
neutral default only when nothing matches — don't Math.max against the >= 1
floor. Preserve the existing "highest multiplier among matched labels wins"
tie-break (so a bonus+penalty PR is unchanged), which removes ONLY the erroneous
flooring of a lone/all-penalty match:

function selectLabelMultiplier(labels: string[], multipliers: Record<string, number>, fallback: number): number {
  const normalized = new Set(labels.map((label) => label.toLowerCase()));
  const matched = Object.entries(multipliers).flatMap(([label, multiplier]) =>
    normalized.has(label.toLowerCase()) ? [multiplier] : [],
  );
  return matched.length > 0 ? Math.max(...matched) : (fallback || 1);
}

(If the intended multi-label rule is "strictest penalty wins," use Math.min(...matched)
instead — but the key fix either way is that the matched set must NOT be maxed
against the >= 1 fallback, so a sub-1 penalty actually bites.)

Test status

Not covered — and the suite has a telling decoy: test/unit/scoring.test.ts
defines labelMultipliers: { bug: 1.2, refactor: 0.5 } in the repo fixture but
never feeds the refactor label into a preview. Every label assertion uses a
bonus or empty config: bug → 1.2, empty-config fallback → 1,
defaultLabelMultiplier 1.05. So the refactor: 0.5 penalty entry is unexercised
dead decoration and the flooring ships green. A regression test should assert a
PR labeled ["refactor"] yields scoreEstimate.labelMultiplier === 0.5 (and that
the merged-score estimate is correspondingly halved).

Confidence note

High. The flooring is proven by executing the exact formula on the config the
codebase itself ships, the multiplier demonstrably scales the score, sub-1
multipliers are a first-class synced config (schema + registry + ruleset), and the
penalty case is provably untested. It is a core scoring-correctness defect that
systematically over-rewards exactly the low-value-label PRs the penalty exists to
dampen.

Distinct from prior reports

Core scoring math, unrelated to the fractional-underscore constant parse (#992),
the suspended-installation scope (#953), predicted-gate, gate-403, or BYOK.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.slopAI slop and/or attempts to game additional points via manipulation or alt profiles.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions