Skip to content

fix(refraction): forge parser fixes (#3203 1-4, #3204, #3205, #3206, #3212 determinism) - #3217

Merged
squid-protocol merged 1 commit into
mainfrom
fix/refraction-forge-parsers
Sep 19, 2026
Merged

squid-protocol merged 1 commit into
mainfrom
fix/refraction-forge-parsers

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Fixes the refraction tools' own parsers found by the #3120 differential. Tool-side only: no engine file changes, so no golden-master bless and no rosetta impact.

Closes #3204
Closes #3205
Closes #3206
Part of #3203 (defects 1–4; defect 5, the SECTION model, stays open)
Part of #3212 (the determinism half; the snapshot test stays open)

What changed

issue fix
#3204 extract_lineage walks the OPEN operand list and switches mode at each INPUT/OUTPUT/I-O/EXTEND keyword.
#3205 analyze_cobol_intent counts EXEC CICS / EXEC SQL by their opening keywords. The unbounded lazy EXEC….*?END-EXEC\. over the joined file is gone.
#3206 cobol-refractor never writes the target. A patched program goes to <clean room>/00_patched_source/<relpath>, and the forge tools read that copy. patch_lexical_content() is the pure core; patch_lexical_traps() takes an optional dest.
#3212 IR dumps serialise sets sorted (_ir_to_json).
#3203 1–4 Paragraph headers must start in Area A (cols 8–11). END-*/GOBACK/EXIT/CONTINUE are never headers. Sequence fields in cols 1–6 are tolerated. Copybooks resolve across the whole repository, nearest first, and never to a member with a PROGRAM-ID.

One extra defect of the same shape turned up while scoring. After the split on PROCEDURE DIVISION, the rest of the header line ( USING PARM-BUFFER.) read as the entry paragraph on 18 CBSA programs. That is how BANKDATA's A010 came out dead (the #3203 comment). paragraph_headers now skips that line.

refraction_differential.py and cobol_answer_key.py now call the graveyard's own paragraph_headers / find_copybook instead of copies of its old regexes, so they measure the forge as shipped.

Scored against the answer key (#3210), forge before → after

field zopeneditor-sample CBSA
units P 58/75 → 58/58 P 452/729 → 452/452
dead (non-trivial) 17 false → 0 claimed P 5/571 → 5/328
copybook paths R 0/14 → 12/14 P 0/29 · R 0/114 → P 99/99 · R 99/114
inputs P 6/12 → 6/6 —
outputs R 0/6 → 6/6 R 0/1 → 1/1 (BANKDATA)

What remains on CBSA is all defect 5. The 228 missing units are all SECTIONs, and all 323 remaining false dead claims are in programs that use sections (fall-through isn't modelled). EXEC counts now match a line-level count on all 31 CBSA programs (BNKMENU 62, XFRFUN 77, BNK1CAC 37).

Recorded in a new update section of docs/refraction_engine_differential.md.

Verification

  • tests/cobol_mainframe: 81 passed. 12 of the 13 new tests fail against origin/main's code; the 13th is an existing test matched by the -k filter.
  • audit_check.py: all clear (ruff, mypy, dead-key, ast-accuracy).
  • End to end: two cobol-refractor runs directly on a CBSA clone left git status clean and gave identical IR dumps and JCL. On a zopeneditor copy with an injected NEXT SENTENCE, the target stayed clean and the patched copy landed in 00_patched_source/COBOL/SAM2.cbl.

Found, not fixed here

Clean-room outputs are keyed by file stem, so zopeneditor's multiroot/sam/SAM2.cbl overwrites COBOL/SAM2.cbl's IR dump, JCL and schema. This predates this PR (00_patched_source mirrors relative paths, so it isn't affected).

🤖 Generated with Claude Code

… EXEC counts, clean-room lexical patch, graveyard Area A + repo-wide copybooks

- #3204: extract_lineage walks the OPEN operand list and switches mode at
  each INPUT/OUTPUT/I-O/EXTEND keyword (was: first mode for every file, so
  no program ever had an output and every DD was DISP=SHR).
- #3205: analyze_cobol_intent counts EXEC CICS / EXEC SQL by their opening
  keywords; drops the unbounded lazy `EXEC ... .*?END-EXEC\.` over the joined
  file, which merged period-less blocks and was a ReDoS shape.
- #3206: the refractor never writes the target repository. A lexically
  patched program goes to <clean room>/00_patched_source/<relpath> and the
  forge tools read that copy. patch_lexical_content() is the pure core;
  patch_lexical_traps() takes an optional dest.
- #3212 (determinism half): IR dumps serialise sets sorted.
- #3203 defects 1-4: paragraph headers must start in Area A, scope
  terminators are never headers, sequence fields in cols 1-6 are tolerated,
  the PROCEDURE DIVISION USING operand is not the entry; copybooks resolve
  repo-wide, nearest first, never to a member with a PROGRAM-ID.
- Harnesses (refraction_differential, cobol_answer_key) call the graveyard's
  own paragraph_headers/find_copybook instead of copies of its old regexes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment