Repository navigation
risk formulas: replace language tiers with a measured fidelity table + a language-strictness table (child of #2560) #2716
Description
Design pass for #2653, approved 2026-09-04. Full write-up (measurements, retractions, the three-input model): the "Language Confidence Tiers" memo, reproduced in summary here so the issue stands alone.
Where we are
keyword-rosetta shows identical §2 raw signals producing different §3 risk scores by language. Most of that is signal_processor._get_tier: two inline hand lists pick fc / irc / ot per language, and 45 of 59 defined languages fall through to the harshest setting (fc 0.60 / irc 5 / ot 1.40).
Measured on language-crucible (2,649 scored files) by scanning once as committed and once with TIER_VARS flattened — baseline reproduces golden_master_zero_dep_audit.json with zero diffs:
- 78.3% of files change; 18.2% by ≥ 25 points; 11.0% by ≥ 50. Tech Debt +19.5 mean on the 31.7% it moves (max 54.9); Documentation +14.7 on 21.2% (max 81.5); Testing max +77.6.
- On rosetta with content held identical:
risk_documentation+87% andrisk_tech_debt+56% tier 1 → tier 3. embedded_pythonis tier 3 (not the literal"python"):boot.py+77.5 Testing,bldc_driver.py+55.4 Documentation. Mean doc movement +26.6 vs python's +1.0.- The heaviest-corrected languages are declarative: makefile +29.3 debt, jcl +29.1, m4 +14.9, html +9.7 doc. They have no runtime.
Why one dial can't be made right
fc / irc / ot is one three-valued lookup doing three separable jobs:
| job | truth lives in | today |
|---|---|---|
| our rules catch less in some languages (fidelity) | rosetta measures it per language × signal | guessed by bucket; 21 of 45 tier-3 languages have full defence-rule coverage and rosetta catches their plants, yet lose 40% of credit |
| this file does runtime-dynamic things a regex can't follow | the file — high_risk_execution, reflection_metaprogramming, eval are already signals |
flat irc by language; 80% of python files and 84% of groovy files contain zero dynamic execution and pay it anyway |
| this language lets you leave things unsaid (types, errors, memory, globals) | the language spec | two undocumented inline sets + a dead FIDELITY_TIERS that disagrees with them for ~20 languages |
Rosetta cannot validate any §3 correction — its null hypothesis is "no correction" — so it corrects the measurement layer only. That is the reason this drifted unnoticed.
The plan
Same equations. The constants they read come from different places.
fcbecomes a measured, per-signal fidelity coefficient generated from rosetta:fc(lang, signal) = min(1, planted / measured). Over-firing rules get their credit scaled to the planted truth; under-firing is a to-fix list, not a coefficient (a coefficient that compensates for a fixable gap stops the fixing). Ledgered n/a → 1.0. Retiressystems_buffer_ratio, whosefc < 1.0gate is the cause of the safety sign inversion.irc/otbecome a documented language-strictness table inanalysis_lens: objective yes/no columns (static types · enforced errors · memory safe · no implicit globals), one row per language, family map soembedded_python→python. No-runtime languages getNone→ no language-level term. Weight small and bounded (UEF small-file normalization isn't achieving its intent — redesign scoring so identical intent scores identically regardless of file size (design pass for Fable) #2655's invariant stays: corrects measured risk, never creates it) until the outcome pilot says otherwise. DeletesFIDELITY_TIERS.- Per-file dynamism replaces the flat
ircin the six formulas that read it, one formula at a time, each with its own crucible before/after. - Outcome pilot: the crucible is snapshot copies (
PROVENANCE.json, 218 repos pinned by SHA, no.git), so clone ~10 upstream at their pins with history, classify fix commits, and ask whether strictness / dynamism predict later fixes at equal evidence. Decides how much weight (2) may ever carry. - Docs: 08-03 rewritten as a chain — what a regex can't see → the three inputs and their sources → the fixture that scores it → the UEF small-file normalization isn't achieving its intent — redesign scoring so identical intent scores identically regardless of file size (design pass for Fable) #2655 invariant — with per-language tables generated, never typed.
What "done" looks like: identical §2 gives near-identical §3 across languages; the only language-level offset left is the small documented strictness term, which rosetta prints rather than subtracts out with the F.3 within-tier band.
Children
- bug: safety
systems_buffer_ratiosign inversion — _calc_safety: systems_buffer_ratio keyed on fc<1.0 makes tier-2/3 files score SAFER than tier 1 above 6/15 attack hits #2717 - engine PR 1: strictness table + fidelity coefficient replace
_get_tier— engine PR 1: language-strictness table + measured fidelity coefficient replace _get_tier (child of #2716) #2718 - engine PR 2: per-file dynamism replaces flat
irc— engine PR 2: per-file dynamism count replaces the flat irc term (child of #2716) #2719 (and the shell reflection over-match it exposed, shell: reflection_metaprogramming matches plain ${var} expansion (indirection marker is optional) and every $(...) -- 1,019 hits in 2,823 LOC on configure-helper.sh #2722 → fix(core-engine): shell reflection counts dispatch, not command substitution (#2722) #2725) - docs: 08-03 chain rewrite (rides with PR 1)
- keyword-rosetta: retire the F.3 within-tier band once PR 1 lands; print the strictness offset instead — done via keyword-rosetta#51, which re-points the band onto
strictness_constants()strata rather than removing banding (2026-09-05 decision)
Spun out: the outcome-label pilot (does strictness/dynamism predict later fix commits?) is tracked independently as #2720 and no longer blocks this epic's closure — it decides future weighting, not whether this replacement is correct.
Related: #2653 (narrowed to the membership audit; closed by PR 1), #2669 (rosetta epic), #2655 (evidence-mass floor and the invariant), #2705.
Activity
squid-protocol commentedon Sep 4, 2026
PR 1 is up: #2721 (closes #2718, #2717, #2653). Two things worth noting from the gauntlet: (1)
rosetta_auditpasses 46/46 with 0 regressions — the corpus gates §2 raw signals, which PR 1 does not touch, so no corpus re-bless is owed; therisk_*rows of the bias report will move on the nextbias-historyregen and keyword-rosetta#50's within-tier banding can then be retired (checklist item). (2) The golden-master attribution came out exactly as the design predicted: 0 topological diffs, zero movement for rust/java/yaml/json, ex-tier-3 languages down, ts/js/python/c/cpp slightly up from the #2717 buffer removal. Fidelity cells the engine actually reads below 1.0: c test 0.67, embedded_python safety 0.50 / test 0.67, haskell safety 0.67, perl test 0.67, solidity ownership 0.25, typescript safety 0.67 — each is a rule that over-fires on plant and is now a to-fix list as well as a coefficient.squid-protocol commentedon Sep 4, 2026
PR 2 is up: #2723 (closes #2719). Dynamism = the file's
reflection_metaprogrammingcount — the first cut also summedhigh_risk_execution, and attribution showed that signal is the safety attack vocabulary across the registry (panic!,System.exit,STOP RUN,rm -rf), so it stays in safety at ×4. After this PR the language-levelircis read by exactly two formulas (safety, tech debt), each with a written reason. The per-file term also surfaced a rule over-match the flat constant had hidden: shell's reflection rule counts every$(...)and plain${var}— #2722, 15 of 46 crucible shell files jump ≥ 25 on Documentation until it is fixed. Suite, audit_check, tree-sitter 30/30, tri 3/3, rosetta 46/46 clean; no corpus re-bless owed.fix(core-engine): shell reflection counts dispatch, not command subst…
squid-protocol commentedon Sep 5, 2026
Status 2026-09-05: is the done-criterion met?
The epic's "done" is: identical §2 gives near-identical §3 across languages, and the only language-level offset left is the small, documented strictness term. Three PRs are in (#2721, #2723, #2725), so this measures that claim directly rather than inferring it from the bias report.
Method. keyword-rosetta corpus (46 languages, 184 shell files) scanned twice with engine main
0e521e8d, full-precision venv: once as shipped, once withSTRICTNESS_IRC_PER_GAP = 0/STRICTNESS_OT_PER_GAP = 0(Irc 0, Ot 1.0 for every language; nothing else touched). Per file,on − offis exactly the strictness term.off − median(off)is everything else: §2 raw signals, engine post-processing, per-file dynamism.main.asm, 27 LOC)b.agc)Documentation: zero strictness, the spread is two input effects
Every point of the 71-pt spread survives with strictness off, so none of it is the language constant. It is:
raw_arch_api 3in a/b/c but scoring readsapi 6; jcl and objective-c goraw 0 → api 3. Doc scores 77–79 on those files vs 38–41 everywhere else, which is the +24 to +29 residual for those languages. Their manifests pinapi_orphan_credit 3deliberately: jcl has no api rule, objective-c's rule matches interface declarations rather than method bodies, m4'sAC_SUBST([PROBE_X])is a different identifier from the macro, and cobol/scheme plant the name (ENTRY 'PROBE-GLOBALS',(export probe-globals)) but the fix(core-engine): the api rule's own declarations are not new orphan surface (#2731) #2734 name test tokenises on\w+, so a hyphenated identifier splits and can never match itself. Ledger:api-declared-orphans-are-not-new-surface(the 25 languages where the credit resolves),api-export-names-suppress-orphan-credit,api-double-count-inflates-scored-api. Nothing here is a risk formulas: replace language tiers with a measured fidelity table + a language-strictness table (child of #2560) #2716 item; the hyphenated-identifier case is the one narrow residual of arch_api counts a declared-public, uncalled function twice: 3 functions record api 6 #2731 if anyone wants it.b.sh+14.2 for twoeval(40.6 → 54.8; +3.5 on the language mean); powershellmain.ps1+1.1 foriex $payload; apexType.forNameand fortranEQUIVALENCE+1 to +4 on the file. One rule worth a look: yacc'sreflection_metaprogrammingis%\{|%\}|%%, which fires on the%%section separator of every yacc file, so every yacc file carries dynamism ≥ 1 by grammar syntax rather than by anything runtime-decided.Tech debt and safety: the offset is documented, and "small" depends on file size
The stratum-mean offsets (+4 to +9 tech debt, +1 to +5 safety) are what keyword-rosetta prints as the documented per-stratum medians. But per file the tech-debt term reaches +35 for every irc3
main.*(37.8 → 73.1) and +44 for assembly. The arithmetic reproduces exactly:irc × 0.5stress on the 50-LOC evidence-mass floor isircdensity points, landing on a sigmoid of slope 0.5 centred at threshold 5 — the steepest part of the curve. The same file at 300 LOC would move +0.8 / +1.6 / +2.5 / +3.5 for irc 1–4. So the term is at its maximum by construction on rosetta's sub-floor files and near-negligible on crucible-sized ones; the #2655 invariant (zero evidence → zero) holds throughout (everya/bfile with no debt scores 0 in both runs).The residual tech-debt spread is the orphan detector, not the language:
main.*records 4 orphans in abap/cobol/css/jcl/m4/objective-c/scheme (→ 99.6), 1 in java/python/ruby/shell/powershell (→ 37.8), 0 in ada/livecode/haskell/sqlite (→ 0); dockerfile's two duplicates put it at 100.Verdict on the checklist
irc0..irc4) and the per-stratum medians are printed as the offset (keyword-rosetta dbc7acd). Decision: no bands are retired; that item is satisfied by the re-pointing.Corrected 2026-09-05: the first version of this comment listed shell among the api-credit languages; that came from a local corpus checkout behind main (main's shell plants are
export -f probe_x, so the credit resolves and shell reads api 3). All other numbers were re-measured against main's data and are unchanged.Reproduction: patch the two constants to 0 via a
sitecustomize.pyonPYTHONPATH, rungalaxyscope <keyword-rosetta>/data/<lang> --db-onlyper language in both conditions, difffile_dataper file.squid-protocol commentedon Sep 14, 2026
All engine/docs children shipped (#2717, #2718→#2721, #2719→#2723, shell fix #2722→#2725, docs). The F.3 within-tier band retirement is satisfied by keyword-rosetta#51 (re-pointed onto strictness strata per the 2026-09-05 decision). The outcome-label pilot (#2720) is spun out as an independent follow-on — it governs future weighting of the strictness term, not the correctness of this replacement — so it no longer blocks closing this epic.
squid-protocol commentedon Oct 23, 2026
Pooled memo for #2716 (outcome-label pilot, #2720)
Pilot: do the language-strictness term and the per-file dynamism term predict later fix commits at equal measured evidence? Method, per-slice results, label checks and limits:
docs/research/outcome_label_pilot.md; reproduce withtests/tools/outcome_pilot.py(clone,scan,label,recall,analyze). Estimator: Mantel-Haenszel rate ratio over repo x LOC decile x evidence quartile strata, stratified-bootstrap 95% CI.hits on 761 files (954 on 526 with the bulk guard), 24-month window capped by HEAD (about 6-7 months for the nine 2026 pins).
micropython 2.3); the contrast is confounded with file role. It supports not raising the weight, and does not by itself prove zero; the call on setting the engine PR 2: per-file dynamism count replaces the flat irc term (child of #2716) #2719 weight to zero is the owner's.
Supports keeping the per-file term at its present modest weight; carried by curl and redis.
Limits: subject-only labels (precision about 83%, recall well below); about 6-7 months of follow-up for nine repos; a rate per file rather than per line; one language row per repo for most repos; dynamism comes from one signal (
reflection_metaprogramming); evidence is a plain sum, not the formulas' weighted view.Decision for the owner: keep the strictness weight as is, or set it to zero in #2719 (the language table then stays as documentation). The pilot supports not raising it; it does not by itself prove zero. The per-file dynamism term can stay at its present weight.
outcome pilot memo: record the posted #2716 comment (Part of #2720)
outcome pilot memo: record the posted #2716 comment (Part of #2720) (#…