Mainframe byte level: raw EBCDIC code pages (037/1047/277/278/930/939, FB80 splitting) and DBCS COBOL (PIC N/G, SO/SI)

Part of epic #3811 -- see its plan. 🤖 Generated with Claude Code

Scoped slice for a separate Claude session: Western Europe single-byte EBCDIC.

Posted 2026-09-27 by the session driving the milestone region by region. Western Europe is the first region we're certifying. Another local session is on #3810 (the Nordic torture estate) and a subagent is on #3828 (DB2 EUR dates / compiler options), so please stay clear of those. Double-byte work (cp930/939, PIC N/G, SO/SI) is out of scope here; it's the East Asia project.

What exists: gitgalaxy/core/source_text.py (#3813) decodes each file with a fixed ladder: BOM → UTF-16 heuristic → strict UTF-8 → the estate's declared code page (--source-encoding or a per-glob map) → cp1252 → Latin-1. java_target.zoned_sign_characters derives the overpunch sign characters for a code page (#3826).

What's missing for Western European estates shipped as raw EBCDIC:

Codecs for the national EBCDIC pages Python lacks: cp277 (DK/NO), cp278 (FI/SE), cp280 (IT), cp284 (ES), cp285 (UK), cp297 (FR) and cp1047 (Open Systems Latin-1).
- Declaring any of these today raises LookupError, and java_target._NATIONAL_ZERO_SIGNS hand-covers the gap.
- Suggestion: register codecs charmap tables generated once from the JDK's charsets (Charset.forName("IBM277") etc.) and committed with provenance. Compare them with the cp037/cp273/cp500 tables Python ships.
- The overpunch table and the equivalence harness's decode_field then work for every page.
Recognising raw EBCDIC. A file of 0x40 spaces and 0xC1-0xE9 letters currently falls through to Latin-1 and reads as garbage. Detect it (a byte histogram) only when the estate declares an EBCDIC page or --source-encoding ebcdic-auto, and record how.
Record splitting for FB80 / FB-n files with no newlines (a raw download of a PDS member), so fixed-format columns stay right. Take LRECL from the declaration or infer 80.
Tests: Unicode Gauntlet cells for the mainframe seed encoded in cp037 / cp273 / cp277 / cp278 / cp297 / cp1047 (the gauntlet's encoding dimension), plus round-trip tests for each new codec.

When done, run python tests/tools/pr_gates.py before the PR (golden masters must stay byte-identical on the existing corpora) and add the PR to milestone "GitGalaxy Tours the Globe".

Update 2026-09-27: the Western Europe single-byte slice (national EBCDIC codecs cp277/278/280/284/285/297/1047, FB-record splitting, the overpunch table from real codecs) is now being done by the local milestone session, not a separate Claude. Gauntlet EBCDIC cells will build on #3881's declared-encoding plumbing once it merges.

#3816: register() needs no global flag; drop an unused import (CodeQL)

#3816 (part 1): the Western European EBCDIC code pages, and raw EBCDI…

Part 2 scoped: gauntlet cells for raw EBCDIC (for the session that did #3881).

Part 1 (#3888, merged) added gitgalaxy/core/ebcdic_codecs.py, which registers cp277 / cp278 / cp280 / cp284 / cp285 / cp297 / cp1047. In source_text, a file decoded with a declared EBCDIC page now ends lines at NEL (0x15), and a file with no line ends at all is split into its 80-byte records. I checked a raw cp277 FB80 copy of the Nordic mainframe seed by hand: it scans with zero fact differences from its UTF-8 twin, run with --source-encoding cp277. The gauntlet doesn't cover this yet.

To do:

Encodings. Add EBCDIC encodings to the Unicode Gauntlet for the mainframe seed only: cp037, cp273, cp277, cp278, cp297 and cp1047. Write each file as fixed-block card images, every line padded to 80 characters, with no line ends. Each cell declares its code page through #3878 (part 1): the Unicode Gauntlet declares Shift-JIS / GB18030, as a real estate does #3881's declared-encoding plumbing.
- The seed's lines must fit in 80 columns; check this.
- A national-letter name must be encodable in the page. ÆØÅ fits cp277 but not cp037, and in that case the cell is skipped, as encode() already does for cp1252.
Scripts per page. Choose them so each page is tested with its own letters: nordic × cp277 / cp278, german × cp273. An ASCII seed goes in every page.
Tests. Cover the FB80 writer and add one cell per page to the default sampled plan, so CI exercises it and not only --full.
Baseline. Should stay empty for these cells. If one fails, it's a real finding: fix it or ledger it with an issue.

Out of scope: double-byte work (cp930 / cp939, PIC N / G, SO/SI). It's the natural follow-on to #3878, as a separate part.

Other work in flight, to avoid collisions: #3834 (the Cultural Gauntlet, main session), #3833 (the ETL unpacker and porting rules), #3815 (Java names), #3867 (.inc detection, which also touches baseline.txt). GitHub ignores the merge=union driver on baseline.txt, so merge main locally before pushing.

#3816 (part 2): an EBCDIC estate keeps its reference's other languages

#3816 (part 2): the Unicode Gauntlet writes the mainframe seed as raw…

#3816 (part 3a): keep the in-suite gauntlet oracle to one EBCDIC page…

#3816 (part 3a): scan Japanese, Chinese and Korean mainframe source (…

#3816: Optional[str], not str | None, in cobol_schema_forge (no __fut…

#3816 (part 3): PIC N / G, USAGE NATIONAL / DISPLAY-1 recorded and si…

All parts merged:

#3816 (part 1): the Western European EBCDIC code pages, and raw EBCDIC sources #3888 (part 1): the Western European single-byte EBCDIC pages (cp277 / 278 / 280 / 284 / 285 / 297 / 1047), and raw EBCDIC sources: NEL line ends, FB80 record splitting.
#3816 (part 2): the Unicode Gauntlet writes the mainframe seed as raw fixed-block EBCDIC #3899 (part 2): the Unicode Gauntlet writes the mainframe seed as raw fixed-block EBCDIC.
#3816 (part 3a): scan Japanese, Chinese and Korean mainframe source (EBCDIC cp930/939/935/937/933) #3901 (part 3a): the East Asian mixed-byte pages cp930 / 939 / 935 / 937 / 933, with SO/SI.
#3816 (part 3): PIC N / G, USAGE NATIONAL / DISPLAY-1 recorded and sized #3926 (part 3): PIC N / G and USAGE NATIONAL / DISPLAY-1 are recorded (NATIONAL was dropped before), sized at 2 bytes per position, typed NVARCHAR, and read as UTF-16BE (NATIONAL) or through a declared data.dbcs_code_page (DISPLAY-1) in the generated Java.

Related, still open: #3908 / #3909 (the declared page must also reach the refactor step, the port tickets and the generated decoder; in progress).

🤖 Generated with Claude Code

fix: train -- #4329, #4391, #4392, #4393 (#4398)

COBOL engine: a group's USAGE applies to its elementary items (#4525) (…

Navigation Menu

Repository navigation

Mainframe byte level: raw EBCDIC code pages (037/1047/277/278/930/939, FB80 splitting) and DBCS COBOL (PIC N/G, SO/SI) #3816

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Mainframe byte level: raw EBCDIC code pages (037/1047/277/278/930/939, FB80 splitting) and DBCS COBOL (PIC N/G, SO/SI) #3816

Description

Activity

squid-protocol commented on Sep 27, 2026

squid-protocol commented on Sep 27, 2026

squid-protocol commented on Sep 27, 2026

squid-protocol commented on Sep 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions