Skip to content

Epic: detect project-local idiom wrappers as a fact channel (follow-up to #3003) #3313

Closed

Description

@squid-protocol

Follow-up to #3003, which documents the limitation in docs/known_blind_spots.md (Blind spot 1).

Problem

Literal-vocabulary rules count calls to a primitive by name. When a project wraps the primitive in its own helper, the rule goes blind to every call site of the helper:

  • curl 8.17 → 8.18: summed state_memory_alloc falls 1197 → 171 (−86%) across the curlx allocator refactor, while LOC grows (temporal-crucible walk panel, 2026-09-13).
  • CPython, reproduced in the pinned crucible (v1.4.0): all 7 CPython C files record Manual Memory Allocation = 0, yet they make 45 calls through PyMem_*/PyObject_*.

In both cases the wrapper is defined in a different file from its callers, so no same-file heuristic can see it.

Design (decided on #3003)

Wrappers are a fact, not a count. The literal signal keeps its syntactic, cross-language meaning, and wrapper knowledge rides alongside it as a named relation, the same signal/fact split the fact channels use (#3200, #3201, #3246, #3250). An opt-in per-project alias list was considered and rejected: it goes unfilled, goes stale, and makes two scans of the same code disagree.

Priority and scope (after steps 0-2; see the comments below)

Medium priority, and memory_alloc is in. The crucible census (step 0) found the in-sample gap concentrated in error and logging helpers (debug_prints, panics_and_aborts). The full-clone census (step 2) found the allocator gap is large and detectable:

  • curl: its allocator layer is function-like #define aliases. Literal calls fall 82% across 8.17 → 8.18; literal plus aliases moves +4%.
  • CPython: its allocator API is function wrappers, with about 10× more wrapper calls than literal ones.

Sub-issues (to file as each starts)

  1. Census: how common are thin wrappers? Done. Results are in the comment below.

  2. Detection heuristic plus a measured precision bar. Done (#3313 step 1: thin-wrapper detection heuristic with a measured precision bar (debug_prints, panics_and_aborts) #3315, PR wrapper_probe: thin-wrapper detection heuristic with a measured precision bar (#3315) #3316). The result is ≤8 LOC with an aligned span and no branches, for debug_prints/panics_and_aborts. The original plan for this step follows.

    • Define a thin wrapper: a function whose body is short (bounded statement/LOC count) and whose substantive content is a call that hits a literal-vocabulary rule.
    • Hand-check precision on the census's real cases: fortran/wrf wrf_error_fatal/wrf_message, numpy outmess/errmess, sqlite ErrorMsg.
    • Measure false attribution: the go/core lock case, where a same-named function outside the sample receives the calls. Call-site credit needs a resolution rule: same file first, then a unique repo-wide definition, and never a qualified x.f(/x::f( call.
    • Decide whether test-suite helpers (Lua checkmessage/RUN) count, and settle the thick-wrapper cutoff.
  3. Full-clone allocator census. Done (#3313 step 2: full-clone allocator census (CPython, curl 8.17/8.18) -- does memory_alloc need wrapper detection? #3324, PR wrapper_probe: --repo mode and #define alias census; full-clone allocator census (#3324) #3326). The gap is material and detectable, with macro aliases.

  4. Fact channel wrapper_data (global/resolved shape, like Mainframe call graph: no edges for COBOL CALL, CICS LINK/XCTL or JCL EXEC PGM=, and unresolved CALL targets are recorded nowhere #3200's call graph). Detection, as measured in steps 1 and 2:

    • Function wrappers: a per-rule filter, with no-branch for debug_prints/panics_and_aborts but not for memory_alloc, since allocator wrappers guard size and NULL.
    • Function-like #define aliases in C-preprocessor languages.
    • A bounded closure across both kinds.
    • Resolution: same-file first, then a unique definition; never a method or a qualified call. For macros, any #ifdef definition counts.

    Record wrapper name → kind (function | macro) → primitive or via → rule → defining file, following gitgalaxy/core/how_to_add_a_fact_channel.md and the add-fact-channel skill.

  5. Derived wrapper-aware count, reported beside the literal signal and never folded into it. Example: debug_prints 160 + 278 calls through 2 detected wrappers (the fortran/wrf case).

    • Needs a contract entry stating it is derived, not a structural signal.
    • Decide whether any score may read it; the default is no.
  6. Validation.

    • fortran/wrf's debug_prints/panics_and_aborts in the crucible.
    • The curl 8.17 → 8.18 walk: the literal signal still drops, and the derived figure should show the surface moving behind curlx_* rather than vanishing.

Out of scope (stays in known_blind_spots.md)

  • Macro wrappers (#define xmalloc malloc).
  • Thick wrappers.
  • Library delegation (curl's def_auth: the primitive never appears in the repo).

Activity

  1. added
    enhancementNew feature, sensor, or structural signature
    epicLarge-scale architectural milestone or multi-issue initiative
    core-engineModifications to the central physics and parsing engine
    on Sep 23, 2026
  2. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Step 1 evidence: thin-wrapper census over language-crucible v1.4.0 (2026-09-22)

    A read-only census, with no engine change: for every crucible repo group, find thin wrappers and count their call sites against the rule's literal hits.

    • Thin wrapper: an extracted function of ≤10 LOC whose body hits a literal-vocabulary rule. A hit that is the function's own name doesn't count.
    • Rules: memory_alloc, debug_prints, telemetry, io, panics_and_aborts, sync_locks, high_risk_execution, regex_execution, serialization_parsing, concurrency, time_date_logic, thread_sleeps, ipc_rpc_bridges, auth_middleware.
    • Precision filters: the wrapper name is defined exactly once in the repo, qualified calls (x.f(, x::f(, x->f() don't count, and names are ≥3 characters. A first pass without these filters was dominated by name collisions (Lua test f/foo, constructor, __init__, Rust X::new().

    How common

    • 58 of 235 repo groups (25%) have at least one called thin wrapper.
    • 9 rule × repo pairs, in 4 repos, are material (≥20 wrapped calls and ≥50% of the literal count).
    rule literal hits calls through wrappers hidden share
    debug_prints 2,395 597 25%
    high_risk_execution 677 394 58% (dominated by one Fortran wrapper)
    panics_and_aborts 2,276 391 17%
    regex_execution 1,963 388 20% (mostly Lua test helpers)
    sync_locks 318 238 75% (inflated by a false attribution, below)
    io 3,620 240 7%
    memory_alloc 1,326 101 8%

    The material cases, each read by hand

    • fortran/wrf (real, strongest): wrf_error_fatal is print *,string followed by stop, with 166 call sites; wrf_message has 112. Its debug_prints go 278 through wrappers vs 160 direct; panics_and_aborts 169 vs 47.
    • python/numpy (real): outmess/errmess wrap sys stream writes. io is 99 wrapped vs 55 direct.
    • c/sqlite (real, below the bar): ErrorMsg wraps fprintf, 64 vs 297 direct.
    • lua/cosmopolitan (real, but test code): checkmessage and RUN are test-suite helpers wrapping string.find and os.execute. Whether assertion-helper calls should count as regex or exec use is debatable.
    • go/core (false attribution): the 106 unqualified lock( calls do not reach connReader.lock() (a method, always called cr.lock()). They go to other lock functions defined outside the sample.

    Findings

    1. The measured gap is error and logging wrappers, not allocators. "Print and die" and "log a message" helpers carry the material cases. Allocator wrapping hides 8% in this corpus.
    2. The census undercounts the allocator case by construction. The crucible samples files per repo, so wrappers defined outside the sample are invisible. CPython's obmalloc.c and curl's curlx are exactly that shape. Sizing allocator wrapping needs full clones.
    3. Deciding which function a call reaches is the precision risk. The go/core false attribution is the name-collision problem any implementation of this epic will face without knowing the receiver or qualifier. A detector must measure precision on hand-checked cases, not trust call-site counts.

    Scope below is updated accordingly.

  3. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Step 1 filed as #3315 (probe + hand-labelled precision set; no engine change).

  4. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Step 1 result (#3315, PR #3316)

    The heuristic is measured against 65 hand-labelled crucible candidates (tests/tools/wrapper_labels.json). The decision re-scores in CI from the committed labels.

    Detection rule for step 3:

    • Thin wrapper: a non-synthetic function of ≤ 8 LOC whose body hits the rule off its own name, with the name in its span's first 3 lines (drops misaligned engine spans) and zero branches (drops check-and-throw helpers).
    • Precision: 77% strict, and 98% of kept call sites belong to real wrappers. No labelled wrapper's call sites are lost.

    Resolution rule:

    • Call sites come from the engine's calls_out_to (unqualified name( only).
    • A call is credited to a same-file definition first, else to a unique repo-wide definition. Ambiguous or out-of-sample names credit nothing.
    • A method (Go receiver, self/cls, qualified header) never receives bare calls. go/core's false lock attribution goes 209 → 0.

    Test helpers are counted, for consistency with the literal rules, which count test code too.

    Residual false positives at the cutoff (recorded in the labels):

    • sprintf matched by debug_prints, a rule quirk.
    • A catch @panic(...) error path inside an expression.
    • print passed as a value.

    Next: step 2, the full-clone allocator census (CPython, curl), which gates memory_alloc support. Step 3, the wrapper_data fact channel for debug_prints/panics_and_aborts, can start in parallel since it doesn't depend on step 2.

  5. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Step 2 result (#3324, PR #3326): memory_alloc is material and detectable, with macros

    Full clones: curl at curl-8_17_0/curl-8_18_0 and CPython v3.14.0.

    clone literal function wrappers (≤12 LOC) #define aliases literal + aliases
    curl 8.17 1,263 555 523 1,786
    curl 8.18 228 14 1,625 1,853
    CPython 3.14 193 1,760 244 437
    • curl's −86% is a rename to macro aliases. curlx_malloc/curlx_free/... are #define curlx_malloc(size) malloc(size) and friends. Literal calls fall 82%; literal plus aliases moves +4%. Function-wrapper detection alone finds 14 sites in 8.18.
    • CPython's API is function wrappers through the allocator table (_PyMem.malloc(ctx, size), which the C rule already matches): PyMem_Free has 1,001 sites and PyMem_Malloc 289. Macros such as PyMem_New chain through them. Wrapper calls outnumber literal calls about 10×.

    Consequences for step 3, now in the scope below:

    1. memory_alloc is in scope.
    2. Function-like #define aliases are in scope for the C-preprocessor languages. known_blind_spots.md's "macro wrappers stay blind" is corrected for this shape.
    3. The filter is per rule: no-branch for debug_prints/panics_and_aborts, but not for allocators. Allocator wrappers guard size and NULL, and the no-branch filter would drop 507 CPython sites.
    4. The closure spans both kinds: a function calling an alias, and an alias calling a function. Without it curl reads 2,341 → 1,867 (literal + function + alias), because its public wrappers moved from free() to curlx_free.
  6. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Correction to the step-2 comment above: #3316 merged just before step 2 was pushed to its branch, so step 2 is in its own PR, #3326. The known_blind_spots.md correction likewise landed after #3314 merged and is in #3325.

  7. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Step 3 is up as PR #3334: the wrapper_data fact channel within the measured scope, plus the memory_alloc label pass (92% of recorded call sites are real allocator aliases, 99% counting destructors). No signal moves (0 differing cells on a full CPython scan); the golden masters gain only the new report block on 24 files.

  8. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Step 4 is up as PR #3337: wrapped_<rule> beside the literal rules (contract in docs/wrapper_aware_count_contract.md; no score reads it). Acceptance: curl, C family, 8.17 → 8.18: literal memory_alloc 1,197 → 171 (−86%, the exact #3003 finding), literal + wrapped 2,629 → 2,719 (+3.4%). The surface moved behind curlx_*; it did not vanish.

  9. squid-protocol commented on Sep 23, 2026

    @squid-protocol
    OwnerAuthor

    Closing: acceptance met

    step what where
    — blind spot documented docs/known_blind_spots.md (#3314, #3325)
    0 crucible census: how common comment above
    1 thin-wrapper heuristic, 65 hand labels, measured filter (≤8 LOC, aligned, no branches: 77% precision / 98% site-weighted) #3315, PR #3316
    2 full-clone allocator census (curl 8.17/8.18, CPython): material and detectable, with macro aliases #3324, PR #3326
    3 wrapper_data fact channel within the measured scope; memory_alloc labels at 92% of call sites (99% with destructors) PR #3334
    4 wrapped_<rule> beside the literal rules, contract doc, no score reads it PR #3337

    Acceptance (curl, C family, 8.17 → 8.18): literal memory_alloc goes 1,197 → 171 (−86%, the #3003 finding exactly), while literal + wrapped goes 2,629 → 2,719 (+3.4%). The surface moved behind curlx_*; it did not vanish.

    Defects and gaps found along the way, filed separately: #3338 (batch has no func_start), #3339 (misaligned engine spans), #3340 (perl debug_prints counts sprintf), #3341 (contract sheet render not gated), #3342 (stale carddemo differential baseline), #3343 (two-label cancelled gate check).

    Not filed, deliberately. These are optional and should come with their own evidence if picked up:

    • allocator wrappers beyond the C family (needs its own census; JS new / Go make are a different vocabulary);
    • labelling the 61 lower-traffic print/abort candidates;
    • letting a score read wrapped_<rule> (a score-contract-audit change, per D1).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    core-engineModifications to the central physics and parsing engineenhancementNew feature, sensor, or structural signatureepicLarge-scale architectural milestone or multi-issue initiative

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions