Skip to content

fix(core-engine): the orphan census counts references, not name repetitions (#2727, #2728) - #2754

Merged
squid-protocol merged 2 commits into
mainfrom
fix/2727-2728-orphan-census
Sep 5, 2026
Merged

squid-protocol merged 2 commits into
mainfrom
fix/2727-2728-orphan-census

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Closes #2727.
Closes #2728.
Part of #2669 (F.4).

orphaned_logic was decided by one line — token_counts[name] <= 1 over the whole code stream —
so whether a function read as orphaned depended on how many times the language's syntax writes its
name, not on whether anything calls it. Three distinct defects fall out of that, two filed and one
found while fixing them.

1. #2727 — syntax that repeats the name masked every orphan

Ada closes with end Probe_Globals;, LiveCode ends a handler by name, Haskell writes the type
signature above the equation. The count is already 2 before anything calls the function, so those
languages could never report one: livecode found 3 orphans across the crucible where the
scoped test finds 127; ada and livecode read 0 per file in keyword-rosetta against a 2.50
baseline.

The fix shape as filed was unsafe, and I corrected it. "Count outside [start_idx, end_idx)"
assumes the span contains the function's declaration. It often doesn't:

  • shell — Mode-B spans start at the {, so a K&R make_module() on the previous line sits
    outside its own body (curl/initscript.sh);
  • ruby — spans are offset outright; 66 of 132 sampled functions have the declaration outside.

Counting naively there makes the declaration itself read as a reference and clears the flag —
measured: ruby 51 → 44 orphans, html 5 → 2, shell 8 → 3, all real ones lost. So when the span
holds no occurrence of the name, the definition site is outside it by construction and exactly one
outside occurrence is discounted. With the correction, no language decreases for that reason.

I also tried span ∪ start_line; it fixes ruby and html but not shell, because start_line points
at the { rather than the line carrying the name — a separate metadata observation worth its own
issue.

The Haskell caveat the issue raised is moot: the span does start at the type signature
(probeGlobals's span begins at probeGlobals :: Int -> Int). Haskell still reads ~0 orphans, but
for a reason the issue didn't anticipate — its module export list names every probe outside every
span. That's keyword-rosetta's already-ledgered api-export-names-suppress-orphan-credit shape,
corpus-side, not an engine gap. The one orphan it does find (entry, unexported and uncalled) is
correct.

2. A third defect neither issue named

\b\w+\b cannot produce a token containing -, : or .. Any name holding one therefore counted
zero and satisfied <= 1 unconditionally — it was an orphan however many times it was called:

language before after name shape
cpp 1054 720 Class::method
tcl 210 148 ::ns::proc
powershell 56 3 Verb-Noun
scheme 60 10 hyphenated define
abap 41 8 hyphenated names
cobol 52 45 hyphenated paragraphs

This also corrects part of #2727's own analysis. Its third table row — abap, cobol, jcl, m4,
makefile, objective-c, scheme at 3.25 orphans/file — is attributed there to names "written once,
invoked by other means". For the hyphenated members the cause was tokenization: the counter
could not represent the name at all. scheme goes 3.25 → 0.25 orphans/file in the corpus on that
alone.

3. #2728 — igniter-keyword buckets entered the checks as ordinary functions

dockerfile's four same-bodied RUN slices scored state_slop_duplicates 1 — the only nonzero
duplicate cell across all 46 corpus languages, and a phantom. css's keyframes slices cleared each
other's orphan flag by repeating the keyword.

The bucket set is derived from each language's own func_start, never hand-listed, so it cannot
drift: _closed_literal_capture() returns names only when the capture is provably a fixed literal
alternation and the empty set otherwise. Exactly three languages qualify — css, dockerfile,
html — and test_closed_literal_capture_is_generated_not_hand_listed pins that membership, so a
new keyword in any of those alternations is picked up automatically and a rule that can produce a
real identifier can never opt in by accident.

Deliberately not added to _UNCOUNTABLE_SLICE_NAMES, as the issue suggested. That set governs
the function population (#2691); removing css's ~150 at-rule slices would take functions_found
to 0 and make every per-function descriptor undefined — #2689's markdown/html shape. That is a real
question, owned by keyword-rosetta's slicer-segments-statements-not-functions entry, and it is not
#2728's measured harm. Recorded in the code rather than silently skipped.

Verification

  • Full suite 7632 passed serially (CI's mode); 5 new detector tests, one per behaviour above.
  • tree_sitter_accuracy_audit --ci --all 30/30 OK; tri_comparison_chart --all --ci 3/3 OK;
    audit_check.py clear; ruff and mypy at baseline.

Golden masters re-blessed: 2278 / 2277 differences. The scoping proof is what moved inside
7. Structural Signatures: Orphaned Logic 581 + Public Exports 71 = 652 of 652. No other
structural signal moved at all, which is exactly what a change confined to the orphan census should
look like. Everything else is that signal's known consumers:

class count
Tech Debt Exposure / tech_debt (orphans are dead weight) 649 + 101
Topological Coordinates X/Y/Z (corpus-wide mass re-solve) 489
API Exposure / Documentation Exposure (via the Contextual Baseline Fix) 98 + 82
Structural Magnitude / Public Exports 71 + 71
group + ecosystem + forensic roll-ups 388

Net crucible orphans 7397 → 7803 (+5.5%): increases where the span fix removes a mask
(typescript +333, livecode +124, php +100, jcl +50, go/rust +46), decreases where the
tokenization fix stops manufacturing them.

Cross-repo

Companion: squid-protocol/keyword-rosetta re-bless, owed after this merges — carries the
rosetta:rebless-owed label per keyword-rosetta/docs/GATING.md's cross-repo flow.

Corpus impact is unusually small for a change this size: 4 cells, api_orphan_credit on
dockerfile/a.dockerfile (1 → 0) and scheme/{a,b,c}.scm (3 → 0). The rosetta manifests do not
gate orphan counts directly — they see them only through the orphan→api conversion — so the
languages whose orphans move but whose api rule already declares them (ada, livecode, haskell)
record no manifest change at all.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WfeHRWra1d6Z5uSReFajSq

squid-protocol and others added 2 commits September 5, 2026 11:44
…itions (#2727, #2728)

`orphaned_logic` was decided by `token_counts[name] <= 1` over the whole code
stream, so whether a function read as orphaned depended on how many times the
language's syntax writes its name -- not on whether anything calls it. Three
distinct defects came out of that one line.

1. #2727 -- syntax that repeats the name masked every orphan. Ada closes with
   `end Probe_Globals;`, LiveCode ends a handler by name, Haskell writes the
   type signature above the equation: the count is already 2 before anything
   calls the function. livecode reported 3 orphans across the crucible where
   the scoped test finds 127; ada and livecode reported 0 per file in
   keyword-rosetta against a 2.50 baseline.

   The test is now scoped to each function's own span, with a correction the
   issue's fix shape missed. The span does NOT reliably contain the
   declaration: shell's Mode-B spans start at the `{`, so a K&R `make_module()`
   on the previous line sits outside its own body, and ruby's spans are offset
   outright (66 of 132 sampled functions). Counting outside the span naively
   makes the declaration itself read as a reference: ruby fell 51 -> 44
   orphans that way, html 5 -> 2, shell 8 -> 3, losing real ones. So when the
   span holds no occurrence of the name, the definition site is outside it by
   construction and exactly one outside occurrence is discounted. With the
   correction no language decreases for this reason.

2. A third defect neither issue named: `\b\w+\b` cannot produce a token
   containing `-`, `:` or `.`, so any name holding one counted ZERO and
   satisfied `<= 1` unconditionally. Every C++ `Class::method`, PowerShell
   `Verb-Noun`, Tcl `::ns::proc`, Scheme hyphenated define and COBOL paragraph
   was an orphan however many times it was called: cpp 1054 -> 720, tcl
   210 -> 148, powershell 56 -> 3, scheme 60 -> 10, abap 41 -> 8. This also
   corrects part of #2727's own analysis -- for the hyphenated members of its
   "name never recurs" row the cause was tokenization, not invocation form.

3. #2728 -- igniter-keyword buckets entered the checks as ordinary functions.
   dockerfile's four same-bodied `RUN` slices scored `state_slop_duplicates` 1,
   the only nonzero duplicate cell across 46 corpus languages and a phantom;
   css's `keyframes` slices cleared each other's orphan flag by repeating the
   keyword. The bucket set is DERIVED from each language's own `func_start`
   (`_closed_literal_capture`), never hand-listed, so it cannot drift: exactly
   css, dockerfile and html have a capture that is provably a closed literal
   set, and a test pins that membership.

   Deliberately NOT added to `_UNCOUNTABLE_SLICE_NAMES`: that set governs the
   function POPULATION (#2691), and removing css's ~150 at-rule slices would
   take `functions_found` to 0 and make every per-function descriptor undefined
   (#2689's markdown/html shape). That is a real question, owned by
   keyword-rosetta's `slicer-segments-statements-not-functions` entry, and it
   is not this issue's measured harm.

Net crucible movement: 7397 -> 7803 orphans (+5.5%).

Both golden masters re-blessed: 2278 / 2277 differences, and the scoping proof
is that inside "Structural Signatures" the only signals that moved are
Orphaned Logic (581) and Public Exports (71) -- 652 of 652. Everything else is
that signal's known consumers: tech debt 750, api/documentation 180, mass and
the corpus-wide topology re-solve 560, roll-ups 388.

Full suite 7632 passed serially; tree-sitter 30/30 OK; tri-comparison 3/3 OK;
audit_check clear.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WfeHRWra1d6Z5uSReFajSq
The local `tri_comparison_chart.py --ci` verification run rewrote 24
`last_reconciled_at` stamps and nothing else (verified: zero non-timestamp
diff lines). The push-to-main companion workflow regenerates this file, so the
churn does not belong in this PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WfeHRWra1d6Z5uSReFajSq
@squid-protocol squid-protocol added the rosetta:rebless-owed Intentionally moves keyword-rosetta counts; audit warns, corpus re-blesses after merge label Sep 5, 2026
@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit f8d1077 into main Sep 5, 2026
43 of 44 checks passed
@squid-protocol
squid-protocol deleted the fix/2727-2728-orphan-census branch September 5, 2026 15:53
squid-protocol added a commit that referenced this pull request Sep 5, 2026
…an-census fix

Main re-blessed for #2754 after this branch's first bless, so the fixtures
conflicted. Merged main (taking its fixtures) and regenerated both from the
merged tree. Scoped against main: 893 differences, 428 topological, 465
substantive -- all jcl files / jcl aggregates plus the two corpus-wide health
averages they feed; zero off-target files. Same shape as the first bless.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
squid-protocol added a commit that referenced this pull request Sep 5, 2026
…ution narrowed to the command executors (#2748–#2751) (#2755)

* feat(core-engine): jcl api, cleanup and globals rules, high_risk_execution narrowed to executors (#2748, #2749, #2750, #2751)

Four rosetta gaps in one language, found by the 2026-09-05 pass over every
non-green cell of the keyword-rosetta bias report and filed as #2748-#2751.

- api: `//name PROC` -- a cataloged or in-stream procedure is the callable
  surface `EXEC name` / `EXEC PROC=name` invokes (api contract, fallback
  family). 13 declarations in 13 crucible files; 185 of 376 EXEC steps call one.
- cleanup: DELETE as a dataset's normal-termination disposition
  (`DISP=(MOD,DELETE,DELETE)`, `(OLD,DELETE)`, `(,DELETE)`) -- JCL's teardown
  idiom. #2610 declined it for the io overlap; #2742 reversed that posture for
  sync_locks and the same narrow-subset measurement applies (36 of 533 DISP=).
  The abend-only positional of an allocation is excluded.
- globals: JOBLIB (vs STEPLIB), a job-level SET symbol (vs a PROC parameter)
  and EXPORT SYMLIST -- the scoped-vs-global distinction the ledger said JCL
  lacked. SET is dual globals+state_mutation, dockerfile's ENV shape.
- high_risk_execution: a bare `PGM=` counted every step (188 of 376 crucible
  EXECs, IEFBR14 and the compilers included). Narrowed to the programs that
  execute caller-supplied commands: IKJEFT01/1A/1B, BPXBATCH/BPXBATSL/BPXBATA2/
  BPXBATA8/AOPBATCH, IRXJCL, SDSF. 61 hits in 48 files after.

20 new strict cases: per-rule partitions, overlap pins (io/sync_locks/state),
an end-to-end deck through prism + splice, ReDoS detonations. Docs: jcl status
sections 1/3/4/6/10 and the api contract's fallback family + audit table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(core-engine): re-bless the golden masters for the jcl rule changes (#2748-#2751)

Both fixtures regenerated with crucible_check.py --update. Every substantive
difference is in jcl files or jcl directory aggregates (zero off-target files);
the rest is the corpus-wide topological re-solve. Field breakdown and direction
of change are in the PR description.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(core-engine): re-bless the golden masters on top of #2754's orphan-census fix

Main re-blessed for #2754 after this branch's first bless, so the fixtures
conflicted. Merged main (taking its fixtures) and regenerated both from the
merged tree. Scoped against main: 893 differences, 428 topological, 465
substantive -- all jcl files / jcl aggregates plus the two corpus-wide health
averages they feed; zero off-target files. Same shape as the first bless.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
squid-protocol added a commit that referenced this pull request Sep 5, 2026
…the TS structural_mass dampener (#2778)

`raw_state_slop_orphans` is the corpus's third-weakest metric (61%). Four
independent bugs each made it under-report; they share one golden-master bless,
and #2768 is unsafe without #2777, so they land together.

#2777 (filed during this work): `_is_orphan` anchored its reference test with
`\b`, which asserts a `\w`<->`\W` transition rather than "no adjacent word
character". A name ending in a non-word character matched NOTHING -- not even
its own declaration -- so the declaration discount fired and the function was
reported an orphan unconditionally. 32 of 32 ruby predicate/bang/setter methods
and 12 of 12 C++ operator overloads in the crucible were dead code regardless of
use. Replaced with lookarounds, exactly equivalent for an all-word name.

#2768: the `len(func_name) > 3` guard was a proxy from the pre-#2727 whole-file
token-frequency test. Against #2754's span-scoped test it only meant a function
named in three characters or fewer could never be reported unused, in any
language -- 3.6% of functions corpus-wide, 29.4% of lua's.

#2774: naming a function in an export statement counted as a use, so a declared-
public uncalled function could never be reported dead -- the exact population
the census exists to find. Fixed with an opt-in `_visibility_export` rule on the
five languages that publish a function by naming it; `_is_orphan` discounts only
the captured name's own span, never a whole line, so a genuine call sharing a
line with an export still counts. The other 41 languages are unchanged by
construction. Not the `api` rule, whose broad visibility modifiers (javascript's
bare `export`) would have swallowed real calls.

#2767: yaml's `func_start` had no capture group, so every extracted step in
every scanned workflow was named `run` -- killing the orphan census, the
duplicate census, and per-step identity. Steps now take their name from the
adjacent `name:` key with a bounded step-over for intervening keys. The keyword
alternation is group 1 and excludes its colon, so `_closed_literal_capture`
recognises it and an unnamed step is censused like dockerfile's `RUN`.

#2769: a missing group made `tsx?$` a top-level alternative in PATH_MODIFIERS,
so every .ts/.tsx file in every scan took a 0.1 structural_mass dampener meant
for icons and assets.

Verified: 7696 passed; rosetta_audit 46/46 clean, 0 regressions; audit_check
clear; golden masters re-blessed with the uncapped diff scoped per field.

Closes #2767
Closes #2768
Closes #2769
Closes #2774
Closes #2777

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
squid-protocol added a commit that referenced this pull request Sep 9, 2026
…#2827) (#2914)

detector.py's overlap test (#2731) decided whether the api rule had already
counted an unreferenced function by tokenising each api-matched line with
\b\w+\b and testing the orphan's name for membership. That tokenizer cannot
produce a token containing `-`, `:`, `.`, `!` or `?`, so every function whose
name holds one -- cobol PROBE-GLOBALS in its own `ENTRY 'PROBE-GLOBALS'`
line, scheme `(export probe-globals)`, lua `M.start`, perl `Pkg::method`,
zig `@"..."`, every powershell Verb-Noun -- was invisible to it, the overlap
read 0, and galaxyscope's Contextual Baseline Fix credited the orphan a
second time: the exact double count #2731 removed, back for every language
whose names are not plain \w. Same tokenizer family as #2754, which fixed
the census and left this consumer behind.

The test now searches the orphan's real name with the census's own
_name_boundary_pattern lookarounds over the api-matched lines, so both
halves of the census agree about what an occurrence of a name is. The "no
false positives" argument in the comment still holds: an orphan's name
occurs exactly once in the file, so a match on an api line is its own
declaration.

Corpus (rosetta_audit against keyword-rosetta origin/main): cobol and
scheme a/b/c api_orphan_credit 3 -> 0, nothing else moves (c's a.c is
#2907's pending re-bless). Label rosetta:rebless-owed. Golden masters
re-blessed in both modes; scoped with a full golden_diff pass: 14 files in
lua (freebsd-src, cosmopolitan), perl (exiftool) and zig (Compilation.zig)
move on exactly Public Exports / API Exposure / Documentation Exposure /
Structural Magnitude, their aggregates follow, the rest is the coordinate
re-solve.

Phase 1 of #2908.

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rosetta:rebless-owed Intentionally moves keyword-rosetta counts; audit warns, corpus re-blesses after merge

Projects

None yet

1 participant