[squid-protocol]

Resolves #3985.

What was actually broken

In a mixed EBCDIC code page (IBM930 / 939 / 933 / 935 / 937), each run of double-byte characters is wrapped in Shift-Out (0x0E) and Shift-In (0x0F). Probed with Java 21 and IBM939:

• Reading was not the defect. A 12-byte field holding AB日本C (9 bytes: C1C2 0E 4562 4566 0F C3) decodes to an 8-character String, and encoding it again gives exactly 12 bytes. The round trip is lossless. The only effect is that length() and substring() stop matching byte positions.
• Writing was the corruption. CobolRecords.putText cut the encoded bytes at the field width. AB日本 put into 6 bytes wrote C1C2 0E 4562 45: a Shift-Out with no Shift-In, and half a double-byte character. It read back as AB日�, and that broken record went back to the mainframe.

Changes

• CobolRecords.fit(value, width, text) cuts to whole characters and pads with spaces to exactly width bytes. It never splits a double-byte character or a surrogate pair, and the JDK encoder closes the Shift-Out run. putText now stores through it.
  • On a single-byte page it writes exactly the bytes it did before; the test checks this against IBM037.
  • This is a deliberate deviation from COBOL's MOVE, which truncates at the byte, as the Javadoc notes.
• CobolRecords.width(value, text) returns a value's width in record bytes, Shift-Out and Shift-In included.
• Porting rule (multibyte_rules), added only when data.code_page can encode a CJK character (cp930–937, shift_jis, cp932, …): fixed-width logic goes through width and fit, never length(), substring() or String.format widths. A reference modification (X(3:5)) of such a field has to be flagged in a TODO, because it can split a double-byte run. Single-byte estates keep today's rules.
• The tickets' target config now includes the data section, so the porter sees the code page. That also covers the codePage argument the Key order: EBCDIC vs the target database's collation -- generated repositories browse and sort differently, MySQL collides case-variant keys #3822 sortKey rule asks for.
• dead_key_audit.py: allowlist code_page. Like decimal_point and key_collation, it's a dataclass field that target_as_dict writes with vars().

Not done, on purpose

• No padding of the decoded String to the byte width (the issue's suggestion). Padding makes length() match, but character positions still aren't byte positions, and encoding the padded String gives more bytes than the field holds.
• No CobolString wrapper. It would change every generated entity's field types for a problem the two helpers already solve.
• No byte-exact refmod helper. A mid-run byte slice has no Shift-Out, so there's no faithful String to return; the ticket flags it instead.

Verification

• test_dbcs_widths.py: a Java test (IBM939 / IBM037 / UTF-8, 18 byte-level cases) covers the lossless round trip, balanced Shift-Out/Shift-In when an over-wide value goes into 4-, 6-, 7- and 8-byte fields, width and fit, IBM037 output unchanged from today, and surrogate pairs. Unit tests check which code pages get the new rule. The test skips without javac; I ran it locally with JDK 21's compiler module.
• tests/cobol_mainframe + tests/cics_crucible: 1170 passed, 22 skipped.
• Ruff, mypy and dead-key audits: nothing new, and the files are ruff format clean.
• Refraction snapshots: I re-ran the snapshot update for the excerpts and all 6 full corpora, and it left no changes. None of those projects generates CobolRecords.
• No parsing logic is touched, so no crucible run was needed.

🤖 Generated with Claude Code

[github-actions]

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab