Skip to content

fix(refraction): graveyard reachability with a SECTION model (#3203) + per-path clean-room keys (#3218) - #3219

Merged
squid-protocol merged 2 commits into
mainfrom
fix/3203-section-model-3218-stem-keys
Sep 19, 2026
Merged

squid-protocol merged 2 commits into
mainfrom
fix/3203-section-model-3218-stem-keys

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Finishes the refraction dead-code finder and stops same-named programs from overwriting each other. Tool-side only: no engine files, no golden-master bless.

Closes #3203 (defect 5; defects 1–4 landed in #3217)
Closes #3218

#3203 defect 5: reachability with sections

x_ray_dead_code used to call a unit live if its name appeared after any PERFORM/GO TO. It now runs a reachability pass over paragraphs and sections, following ranges from the entry:

  • fall-through until a terminal statement (GOBACK, STOP RUN, EXIT PROGRAM, unconditional GO TO, EXEC CICS RETURN/XCTL/ABEND);
  • a transfer inside an unterminated IF/EVALUATE is conditional;
  • PERFORM of a paragraph, of a section (to its last paragraph), and PERFORM … THRU;
  • GO TO, staying inside the enclosing range when its target is in it;
  • a unit ending in a PERFORM of a range that never returns is itself terminal (CBSA's PERFORM GET-ME-OUT-OF-HERE);
  • EXEC CICS HANDLE ABEND/CONDITION/AID labels are entry points.

Other behaviour changes:

  • Comment lines and literals are ignored.
  • The rest of a multi-line PROCEDURE DIVISION USING … header is skipped.
  • The *-EXIT exemption is gone. An EXIT paragraph that nothing reaches is dead; the answer key counts it as trivial dead code.

The DAG architect's and slicer's dead-unit masking now find headers with the graveyard's unit_header. Before, each had its own ^[ \t]{0,7}NAME\., which missed sections and sequence-numbered source, so a dead section's OPENs were never masked.

Answer key (#3210), forge before → after on CBSA: units R 452/680 → 680/680, dead P 57/380 → 62/62, non-trivial dead P 5/328 → 10/10. zopeneditor stays exact (0 dead, 0 claimed).

⚠️ These scores are not independent evidence. cobol_answer_key.py says the drafter is "deliberately NOT shared with, or imported by, either parser it grades". So this is the forge's own implementation, written against its input: upper-cased text with copybooks already inlined. It still follows the same control-flow model as the drafter. The key is hand-verified, but agreement is expected wherever the hand pass left the draft unchanged, and a defect in the shared model would not show up in this score. The unit tests cover each construct separately.

#3218: output keys

  • Output names: clean-room outputs (IR dump, JCL, schema, slice, agent ticket) and the dead-code IR state are keyed by the stem when it is unique in the run. Otherwise the key is the path flattened with __ (multiroot__sam__SAM2). Repos without collisions, like CBSA, keep today's file names.
  • Anomaly tags: these now carry the path under the target. forge_agent_jobs resolves that path directly instead of taking the first rglob(name) hit, and gets ir_keys so it joins the right IR dump.
  • Dropped flags: it also used to drop every [FILE : Line N] system-limit flag, because it rglob-ed the whole FILE : Line N string.
  • SQLite merge proven: test_same_named_programs_do_not_share_outputs_or_dead_code[SQLITE] fails with stem-keyed state. Program B inherits A's dead paragraph, masks its own live OPEN OUTPUT, and loses its only output.

Verification

  • tests/cobol_mainframe: 94 passed (with test_batch_test_harness). The 12 new tests fail against origin/main.
  • Two existing fixtures relied on the old heuristic (a MAIN paragraph with no terminal statement "not reaching" the next one). They now carry a GOBACK.
  • audit_check.py: all clear.
  • The new regexes scale linearly on pathological inputs of 50k–100k tokens (×4 input → ×4 time).
  • End to end: cobol-refractor on both clones left them clean. zopeneditor now writes COBOL__SAM1/2 and multiroot__sam__SAM1/2 separately. CBSA's file names are unchanged, and it reports 62 unreachable units.
  • docs/refraction_engine_differential.md has a new update section.

🤖 Generated with Claude Code

…+ per-path clean-room keys (#3218)

#3203 defect 5: x_ray_dead_code replaces "named in any PERFORM/GO TO" with a
reachability pass over paragraphs AND sections: fall-through until a terminal
statement, PERFORM of a paragraph / a section (to its last paragraph) /
THRU ranges, GO TO, a trailing PERFORM of a never-returning range as
terminal, and CICS HANDLE labels as entry points. Comment lines and literals
are ignored; the *-EXIT exemption is gone. The forge's own implementation of
the answer-key drafter's model -- the drafter stays unimported by any parser.
CBSA against the key: units 680/680, dead 62/62, non-trivial dead 10/10.

The DAG architect's and slicer's dead-unit masking use the graveyard's
unit_header instead of their own `^[ \t]{0,7}NAME\.`, which missed sections
and sequence-numbered source.

#3218: clean-room outputs and the IR dead-code state are keyed by stem when
unique in the run, else by the path flattened with `__`. Anomaly tags carry
the path under the target; forge_agent_jobs resolves it directly (and no
longer drops `[FILE : Line N]` system-limit flags) and takes ir_keys.
In SQLite mode, stem keying merged two same-named programs' dead code and
masked the second one's live OPEN.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

…e literal read as a Shannon-entropy > 4.8 payload)

Same compiled pattern; each literal is now under X-Ray's 64-char extraction
floor, so the file needs no XRAY_BYPASS_PATHS entry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@squid-protocol
squid-protocol merged commit 90b605c into main Sep 19, 2026
29 checks passed
@squid-protocol
squid-protocol deleted the fix/3203-section-model-3218-stem-keys branch September 19, 2026 19:21
squid-protocol added a commit that referenced this pull request Sep 20, 2026
… unexplained count (#3211) (#3243)

* feat(refraction): classify every differential delta by cause, gate on unexplained count (#3211)

`tests/tools/refraction_differential.py` reported raw forge<->engine-DB deltas
and always exited 0; attributing each delta was done by hand (the D1-D4 prose in
docs/refraction_engine_differential.md). This moves the classifiers into the
harness:

- classify() attaches a `cause` to every delta by mechanism (scope_terminator,
  area_b_header, section_header, entry_point, program_inlined_as_copybook,
  ambiguous_copy_target, exec_sql_include, sequence_number_field, system_copybook,
  bms_symbolic_map) or as stated_absence by construction, reusing the graveyard
  finder and answer-key readers rather than new parsers.
- where a corpus has a validated answer key (#3210), an INDEPENDENT-field delta
  (program_id, copybook) is adjudicated to a verdict from the key directly;
  units/dead share the forge's control-flow model (#3219) so their key agreement
  never clears `unexplained`.
- a committed per-corpus baseline (refraction_differential_baseline.json, split
  into excerpts/full scopes) + `--ci` / `--update-baseline` / `--corpus` gate:
  a run that ADDS unexplained deltas exits non-zero. zopeneditor reaches 0
  unexplained; CBSA's 33 (dead-code, gated on #3198) and carddemo's 167 (no key)
  are baselined with a note.

The raw #3120 report is unchanged (repo is now optional; --json gains per-delta
cause/verdict). New tests/cobol_mainframe/test_refraction_differential.py gates
the excerpts in CI and the full corpora when fetched, plus synthetic per-cause
unit tests including the #3219 shared-model guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(refraction): stop leaking scan env into the pytest session; break the answer-key import cycle (#3211)

Two follow-ups from CI on #3243:

- `run_target` set GITGALAXY_DISABLE_GIT_HISTORY / GITGALAXY_LICENSE_KEY on the
  process env with os.environ.setdefault, which persisted for the rest of a
  pytest session and disabled git for a later test (test_chronometer read the
  same var). Scope them to the in-process excerpt scan with a save/restore
  context manager instead; the full corpora already scan in mainframe_corpus's
  own subprocess env.
- CodeQL flagged the cyclic import my classify() introduced. Move the shared
  forge readers old_paragraphs / old_copybooks into cobol_answer_key (beside the
  other COBOL readers) and drop score()'s back-import of refraction_differential,
  so the dependency is one-directional (harness -> reader). Behaviour is
  unchanged, so the committed baseline still holds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant