test(cobol): referee panel -- independent COBOL/CICS parsers scored per fact channel (#4377) by squid-protocol · Pull Request #4384 · squid-protocol/gitgalaxy

squid-protocol · 2026-10-05T00:48:59Z

Part of #4377: a first, measured pass of the COBOL / CICS referee panel.

What this adds (test tooling only)

tests/tools/referees/:

One adapter per referee, each emitting a common referee-facts/1 JSON: Cobrix, Spantree tree-sitter-cobol-enterprise, Eclipse Che4z (its headless analysis CLI) and mapa (CallTree).
The engine (master DB via GalaxyIR) and the estate-crucible key as fact sources.
score.py: the scorecard below, plus a disagreements.json grouped by pattern for adjudication.
A README on installing each referee.

Unit tests cover the scoring logic only (tests/cobol_mainframe/test_referee_score.py). There is no CI workflow yet.

Nothing under gitgalaxy/ changes. No referee code is vendored: Che4z (EPL-2.0) runs as an unmodified external jar, and the only Java here is our own 90-line Cobrix driver.

Engine: ca152eb92b9b (origin/main at run time), zero-dependency install, fresh scans. Referees:

Cobrix 339ddba (2.11.2-SNAPSHOT, Maven build)
tree-sitter-cobol-enterprise 86a2c47 (generated with tree-sitter-cli 0.25.10, py-tree-sitter 0.26)
Che4z 2.5.1 (VSIX server.jar, commit 8624da64)
mapa 6ebe2d4 (committed CallTree.jar)

Corpus:

The four keyed CICS estates at their pins: CardDemo, CBSA, GenApp, zECS. DSF is PL/I and out of scope.
estate-crucible v0.3.0 (checkout verified on the pin), with its generated key converted to the answer-key shape by estate_key.py.

Referee panel scorecard (#4377)

Corpora: aws-mainframe-modernization-carddemo @ 59cc6c2fd7eb, cics-banking-sample-application-cbsa @ 417334533178, cics-genapp @ f6f3f4b2580d, estate-crucible @ 0.3.0, zecs @ 6d6bcbbc89c9.

Parse success and speed (the key-covered members each source reads: programs and / or keyed copybooks)

source	version(s)	clean	partial	failed	median s / member
engine	`ca152eb`	242	0	0	—
che4z	2.5.1	35	70	29	2.487
cobrix	cobol-parser_2.12-2.11.2-SNAPSHOT	92	0	16	0.002
mapa	6ebe2d4059f5	127	0	7	0.995
tree-sitter-cobol-enterprise	86a2c479cb7e	51	73	10	0.002

Per corpus, clean / partial / failed:

source	aws-mainframe-modernization-carddemo	cics-banking-sample-application-cbsa	cics-genapp	estate-crucible	zecs
engine	105 / 0 / 0	63 / 0 / 0	44 / 0 / 0	23 / 0 / 0	7 / 0 / 0
che4z	2 / 19 / 23	16 / 12 / 3	6 / 25 / 0	11 / 9 / 3	0 / 5 / 0
cobrix	55 / 0 / 6	25 / 0 / 7	10 / 0 / 3	—	2 / 0 / 0
mapa	44 / 0 / 0	31 / 0 / 0	31 / 0 / 0	17 / 0 / 6	4 / 0 / 1
tree-sitter-cobol-enterprise	24 / 20 / 0	0 / 31 / 0	15 / 16 / 0	12 / 4 / 7	0 / 2 / 3

Each source vs the answer keys (ground truth), all corpora pooled

channel	engine	che4z	cobrix	mapa	tree-sitter-cobol-enterprise
program_ids	P 100.0% · R 100.0%	P 100.0% · R 73.5% (R parsed 95.2%)	n/a	P 100.0% · R 94.1% (R parsed 99.2%)	P 100.0% · R 91.2% (R parsed 100.0%)
units	P 99.9% · R 100.0%	P 100.0% · R 71.7% (R parsed 98.8%)	n/a	n/a	P 99.8% · R 89.0% (R parsed 99.4%)
unit_extents	P 99.8% · R 99.2%	P 99.5% · R 71.0% (R parsed 98.2%)	n/a	n/a	P 98.8% · R 88.1% (R parsed 98.5%)
edges	P 99.5% · R 99.9%	P 100.0% · R 60.7% (R parsed 99.4%)	n/a	n/a	P 99.2% · R 89.7% (R parsed 99.2%)
calls	P 100.0% · R 100.0%	n/a	n/a	n/a	P 100.0% · R 97.0% (R parsed 100.0%)
call_targets	P 100.0% · R 100.0%	n/a	n/a	P 98.1% · R 71.7% (R parsed 71.7%)	P 100.0% · R 96.6% (R parsed 100.0%)
copybooks	P 100.0% · R 100.0%	P 100.0% · R 68.9% (R parsed 98.7%)	n/a	P 100.0% · R 98.9% (R parsed 100.0%)	P 100.0% · R 97.5% (R parsed 100.0%)
data_items	P 99.9% · R 99.9%	P 99.5% · R 67.5% (R parsed 99.2%)	n/a	n/a	P 99.8% · R 92.2% (R parsed 99.8%)
pic	P 99.8% · R 100.0%	P 99.8% · R 70.1% (R parsed 99.3%)	n/a	n/a	P 99.8% · R 90.6% (R parsed 100.0%)
usage	P 99.9% · R 100.0%	P 100.0% · R 75.1% (R parsed 99.3%)	n/a	n/a	P 100.0% · R 81.6% (R parsed 100.0%)
occurs	P 100.0% · R 100.0%	P 51.4% · R 28.1% (R parsed 50.0%)	n/a	n/a	P 100.0% · R 96.9% (R parsed 100.0%)
redefines	P 100.0% · R 100.0%	P 100.0% · R 63.7% (R parsed 100.0%)	n/a	n/a	P 100.0% · R 97.5% (R parsed 100.0%)
value	P 95.7% · R 98.6%	P 95.2% · R 66.5% (R parsed 98.4%)	n/a	n/a	P 94.7% · R 83.6% (R parsed 98.0%)
layouts	P 100.0% · R 100.0%	n/a	P 100.0% · R 89.5% (R parsed 100.0%)	n/a	n/a
sql_access	P 100.0% · R 100.0%	n/a	n/a	P 100.0% · R 78.3% (R parsed 78.3%)	P 100.0% · R 100.0% (R parsed 100.0%)
cics_commands	P 100.0% · R 100.0%	n/a	n/a	n/a	P 100.0% · R 90.2% (R parsed 99.7%)
cics_files	P 100.0% · R 100.0%	n/a	n/a	P 81.5% · R 84.6% (R parsed 84.6%)	P 100.0% · R 96.2% (R parsed 100.0%)

P = correct / reported, R = correct / true over (file, value) pairs; R parsed counts only the files the source parsed. n/a = the source does not produce that channel.

Agreement with the engine (|A ∩ B| / |A ∪ B| on files both parsed)

channel	che4z	cobrix	mapa	tree-sitter-cobol-enterprise
program_ids	95.2%	n/a	99.2%	100.0%
units	98.6%	n/a	n/a	99.0%
unit_extents	97.3%	n/a	n/a	96.4%
edges	98.8%	n/a	n/a	98.1%
calls	n/a	n/a	n/a	100.0%
call_targets	n/a	n/a	70.7%	100.0%
copybooks	98.7%	n/a	100.0%	100.0%
data_items	99.1%	n/a	n/a	99.9%
pic	99.3%	n/a	n/a	99.9%
usage	99.2%	n/a	n/a	99.9%
occurs	34.0%	n/a	n/a	100.0%
redefines	100.0%	n/a	n/a	100.0%
value	99.5%	n/a	n/a	97.6%
layouts	n/a	99.9%	n/a	n/a
sql_access	n/a	n/a	78.3%	100.0%
cics_commands	n/a	n/a	n/a	99.7%
cics_files	n/a	n/a	71.0%	100.0%

How to read it

The engine vs the keys is near-ceiling, but that is partly circular. The answer keys were adjudicated with the engine as one of the inputs (COBOL modernization answer key: hand-verified per-program truth for the #3120 corpora, scored against forge and engine #3210, Answer keys: score unit extents and intra-program PERFORM / GO TO edges, not just unit names and start lines #4318). The independent signal is the referees: where they parsed, they agree with the engine at 96-100% on every structural channel (units, extents, edges, calls, COPY, data items, PIC / USAGE / REDEFINES, typed EXEC CICS, SQL table access) and at 99.9% on Cobrix layouts.
Where the engine loses:
- It records no extent for the main line (15 main lines in CardDemo / zECS, R 99.2%). This is the synthetic __global_context__ row's design (COBOL engine: statements between PROCEDURE DIVISION and the first paragraph belong to no unit, so the program's main-line PERFORM/CALL edges are never recorded #4302).
- It drops the self-loop GO TO 8500-READTRNX-READ in CardDemo CBSTM03A. This is deliberate (#3327 follow-up: COBOL GO TO is not a call (contract C4) -- needs a decision #3362, engine_drops_self_transfer in Answer keys: score unit extents and intra-program PERFORM / GO TO edges, not just unit names and start lines #4318), but tree-sitter and the key both have the edge.
- It records phantom PERFORM targets (TEST, 3, WS-N) in estate ACCTRPT and CBSA BANKDATA (open: COBOL engine: inline PERFORM forms produce phantom callees (TEST, FOREVER, WS-N of WS-N TIMES, 3, END-IF after EXIT PERFORM) #4305).
- It treats the IDMS IDMS-CONTROL / PROTOCOL / SCHEMA sections as units (open: COBOL engine: IDMS IDMS-CONTROL SECTION / PROTOCOL. (ENVIRONMENT DIVISION) recorded as procedure units #4306).
- Two new findings, drafted as issues below: continued VALUE literals, and a D-line extent.
- The remaining estate-crucible misses (LNCALC / LNRATE PROGRAM-ID, KYUYCPY after a U+3000 space, BILLCALC USAGE) are horrors the estate scorer already reports against COBOL engine: PROGRAM-ID not recorded when its separator period is missing (PROGRAM-ID. CBLDB22 + trailing blanks; PROGRAM-ID TEST16.) #4307 / COBOL: a full-width space (U+3000) separator hides data entries and pictures (opensourcecobol4j dialect) #3956 / COBOL engine: the last DATA DIVISION entry takes its USAGE from the PROCEDURE DIVISION (DISPLAY verb, 'BINARY' literal) -- wrong byte length #4329.
Every referee loses more than the engine on these estates. Their main failure modes, all adjudicated against the source:
- tree-sitter-cobol-enterprise (0.002 s / member, 124 / 134 parsed):
  - CBL CICS(SP) compiler-option cards make the whole program unparseable (3 of 5 zECS programs).
  - ENTRY 'DLITCBL' USING ... is misread, so its last operand becomes a phantom paragraph (3 CardDemo IMS programs).
  - DBCS / Japanese names come out mangled, and EBCDIC members are unreadable (estate).
  - An extent ends at EXIT. instead of covering a procedure-division EXEC SQL INCLUDE. That one is definitional.
  - 73 of 134 members carry ERROR regions, but outside them the facts are 98-100% correct.
- Cobrix (layouts only, 92 / 108 copybooks):
  - It is exact everywhere it parses: 100% precision, and it confirms the engine's offsets and sizes on 2,438 of the 2,439 keyed fields it parsed.
  - It rejects POINTER, level-77 copybooks, EXEC SQL DECLARE (.dcl), lowercase pic x(40) ("field size 0"), and some multi-value 88 lists.
  - Its one wrong answer is pic xXXXXX, read as 5 bytes instead of 6 (CBSA GETSCODE). Cobrix is wrong; the engine and the key are right.
- Che4z 2.5.1 CLI (105 / 134 with facts, about 2.5 s / member plus JVM start-up):
  - The analysis CLI exits 1 with no output on any program containing a CALL statement. A 5-line program reproduces it. This is why 22 of 44 CardDemo programs fail.
  - Its --ast writer overflows the stack or runs out of memory on 2 large CICS programs.
  - The serialised OCCURS clause has no DEPENDING ON, which explains its occurs column (17 of its 35 keyed tables are ODO).
  - Where it produces an AST, it agrees with the engine at 97-100% (units 98.6, edges 98.8, data items 99.1, PIC 99.3).
  - It also chokes on CBL CICS(SP), but recovers.
- mapa (127 / 134, about 1 s / member):
  - It is precise (P 100% on program IDs, COPY, call targets and SQL), but it silently drops facts in CBSA. It reports no EXEC CICS LINK PROGRAM(WS-ABEND-PGM) (an identifier with a VALUE) and misses most DB2 statements and some literal DELETE FILE('CUSTOMER'), so its call-target recall is 72%.
  - It has no ENDBR record type.
  - It reports an unresolved data name (ZF-FCT) as if it were a file (zECS).
  - One member with an uncaught exception aborts its whole run. The adapter now falls back to one run per member.

Adjudicated disagreements (at least 3 sampled per pattern, against the source)

pattern	count	verdict
tree-sitter fails `CBL CICS(SP)` programs (zECS ZECS000/001/003)	3 files	referee wrong (dialect gap)
tree-sitter phantom paragraph `PADFLPCB` / `PAUTBPCB` from `ENTRY ... USING` (DBUNLDGS, PAUDBLOD, PAUDBUNL)	3 units + 10 edges	referee wrong
tree-sitter exit-paragraph extent stops before `EXEC SQL INCLUDE` / `COPY` in PROCEDURE (COTRTLIC 9550, COTRTUPC 9800, COCRDSLC)	13	definitional (key + engine count the copied code)
tree-sitter misses ZECSPLT paragraphs / START / WRITEQ inside an ERROR region	11 units	referee wrong
tree-sitter DBCS names (KYUYO02 `賞与額` → `Z`)	2	referee wrong
tree-sitter / Che4z have a main-line extent, the engine has none (CBACT0xC, ECS001, ZECS00x)	15	engine gap (by design, #4302)
engine `PERFORM TEST` / `3` / `WS-N` (ACCTRPT, BANKDATA A010)	8	engine wrong (open #4305)
engine IDMS sections as units (LNIDMS01)	3	engine wrong (open #4306)
engine misses the self `GO TO 8500-READTRNX-READ` (CBSTM03A)	1	engine wrong, deliberate (#3362)
engine extent CDW010 L1407-1436 vs L1433 (key, tree-sitter, Che4z)	1	engine wrong, new (draft B)
engine VALUE of a continued literal (CBSTM03A HTML-L08 ...)	11	engine wrong, new (draft A); the key is wrong too
engine, tree-sitter and Che4z read `VALUE +18` / `+0` / `-1`; the key has null (lgacdb01 L83, ABNDPROC L40, CBPAUP0C L82)	83	key wrong
key names `10 PIC X(20) VALUE ...` as `PIC` (COPAUS1C L58-67); engine FILLER	10	key wrong
key `VALUE X'0D25'` → `X` (ZECS001 L45, ZECS003 L49)	3	key wrong
key `COMM-OVERDR value 'TO'` (last DATA DIVISION item, BNK1CAC L157 and 9 more)	10	key wrong (key-side twin of #4329)
key strips a literal's leading / trailing blanks (`' LGACDB01'`)	many	definitional (the scorer trims both sides)
Cobrix `pic xXXXXX` = 5 bytes (GETSCODE)	1	referee wrong
Che4z occurs without DEPENDING ON (COPAUS1C L153, COTRTLIC L494, COTRTUPC L341)	17	referee export gap
mapa misses `LINK PROGRAM(WS-ABEND-PGM)` (BNK1CCA, BNK1DAC, BNK1CRA)	41	referee wrong
mapa misses CBSA DB2 access (XFRFUN `SELECT ... FROM ACCOUNT`, DELACC, BANKDATA)	13	referee wrong
mapa `ZF-FCT` as a file name (ZECS000/001)	15	referee wrong (unresolved operand)

Draft issues (not filed)

Draft A (engine)

COBOL engine: a continued nonnumeric literal in a VALUE clause is stored with the raw continuation line instead of the joined literal

Found by the referee panel (#4377). In CardDemo app/cbl/CBSTM03A.CBL, eleven 88-level items carry an HTML literal continued onto a second line (fixed format: the literal runs to column 72, the next line has - in column 7 and resumes after a quote):

157:             88  HTML-L08 VALUE '<table  align="center" frame="box" styl
158:      -             'e="width:70%; font:12px Segoe UI,sans-serif;">'.

record_data.value_literal for HTML-L08 (engine ca152eb92b9b) is

'<table  align="center" frame="box" styl\n      -             '

that is, the first line's fragment plus the raw continuation line up to its opening quote. The value should be the joined literal, <table align="center" frame="box" style="width:70%; font:12px Segoe UI,sans-serif;">. tree-sitter-cobol-enterprise also keeps the raw text, so it cannot referee this. The answer key is wrong too, in a different way: it stores STYL\n (see draft B).

Scope: 11 items, all in CBSTM03A. The other three keyed CICS estates have no continued VALUE literal, and estate-crucible has none either.

Acceptance: the value of a continued alphanumeric / national literal is the concatenation per the Enterprise COBOL continuation rules: the first line's characters through column 72, then the continuation line's characters after its opening quote. Pin HTML-L08 / HTML-L10 in a unit test, and add a horror to estate-crucible.

Draft B (engine, minor)

COBOL engine: a unit's extent runs over trailing column-7 D (debugging) lines when WITH DEBUGGING MODE is off

CBSA BANKDATA.cbl CDW010: the engine's extent is L1407-1436. The answer key, tree-sitter-cobol-enterprise and Che4z all give L1407-1433. Lines 1435-1436 are D lines ( D DISPLAY 'WS-DT-REM' ...), and the program's SOURCE-COMPUTER ... WITH DEBUGGING MODE (line 39) is commented out, so those lines are comments. The engine does not otherwise read D lines as code: CREACC's D MOVEs (lines 522 and 688) have no data_move rows. Only the unit end counts them. #4318's cause list mentions this as part of engine_unit_end_counts_noncode_lines ("the PL/I and D-line parts may deserve their own issue"), but no issue tracks it.

Acceptance: without WITH DEBUGGING MODE, a D line neither starts nor extends a unit. With the mode on, it is code.

The key-quality findings (signed numeric VALUEs, PIC as a name, truncated hex, a VALUE taken from the PROCEDURE DIVISION, continued literals) are in the table above. They are test-tooling fixes for the answer-key owners, not engine bugs.

Recommendation for the next phase

Keep Cobrix as the layout / byte-offset referee. It is cheap (one JVM run per corpus, ms per copybook) and exact where it parses. Its rejections (POINTER, 77, lowercase PIC, .dcl) are easy to report as "not attempted" rather than as disagreements.
Keep tree-sitter-cobol-enterprise as the fast, broad structural referee: units, extents, edges, calls, COPY, data items, typed CICS / SQL, with no JVM and Python bindings. Upstream fixes would help it: CBL / PROCESS cards, ENTRY, DBCS names. It is also the strongest Translator front-end bake-off: which COBOL parser should the det translator use? #4378 front-end candidate seen here.
Che4z: keep, but drive it over LSP (documentSymbol + diagnostics) instead of the 2.5.1 analysis CLI. The CLI's CALL crash and AST-serialisation failures cost it about a quarter of the members. Where it works, it is the most independent industrial referee (preprocessor + copybook resolution).
mapa: keep only for the call-tree / JCL channels, as a secondary opinion. It is precise but silently incomplete on CBSA, and it does not flag what it drops.
Skip ProLeap / cobol-rekt for now. Two structural referees plus Cobrix already over-determine every channel except dynamic call resolution, and neither tool adds dialect coverage.
Next:
- Fix the four key-quality findings, so engine-vs-key numbers stop carrying key noise.
- Add a "referee majority" column (engine vs 2-of-3 referees).
- Run nightly rather than as a CI gate. JVM start-up and the 5-minute grammar generation make it a poor PR gate.

The local audits pass: ruff, mypy and dead_key (--ci), the #3813 source-reads gate, and 16 new unit tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FNWHcUSn5p7eMxFTsB5rFo

…ct channel (#4377) tests/tools/referees/: a common referee-facts/1 format, one adapter per referee (Cobrix, Spantree tree-sitter-cobol-enterprise, Eclipse Che4z's analysis CLI, mapa's CallTree), the engine and estate-crucible keys as fact sources, and score.py, which scores every source against the answer keys and against the engine and groups the disagreements for adjudication. Test tooling only: no referee is vendored, and nothing under gitgalaxy/ changes. Part of #4377. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FNWHcUSn5p7eMxFTsB5rFo

…extents (#4377) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FNWHcUSn5p7eMxFTsB5rFo

+COPY_EXTS = {".cpy", ".copy", ".dcl", ".cbl", ".cob", ".inc", ""}
+_CICS_FILE = {"CICSREAD": "READ", "CICSWRITE": "WRITE", "CICSREWRITE": "REWRITE", "CICSDELETE": "DELETE",
+              "CICSSTARTBR": "STARTBR", "CICSREADNEXT": "READNEXT", "CICSREADPREV": "READPREV"}  # fmt: skip
+_CALL_TYPES = {"CALLBY": "CALL", "CICSLINKBY": "LINK", "CICSXCTLBY": "XCTL", "SQLCALLBY": "SQLCALL"}


github-actions · 2026-10-05T00:56:36Z

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

squid-protocol and others added 2 commits October 4, 2026 20:37

test(cobol): referees -- mapa per-member fallback, FROM lists, Che4z …

bed449a

…extents (#4377) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FNWHcUSn5p7eMxFTsB5rFo

squid-protocol mentioned this pull request Oct 5, 2026

Referee panel: rate GitGalaxy's COBOL / CICS parsing against independent open-source parsers on CICS estates #4377

Closed

github-advanced-security Bot found potential problems Oct 5, 2026

View reviewed changes

squid-protocol marked this pull request as ready for review October 5, 2026 01:21

squid-protocol merged commit f5b5e83 into main Oct 5, 2026
29 checks passed

squid-protocol deleted the feat/4377-referee-panel branch October 5, 2026 01:21

Navigation Menu

Repository navigation

test(cobol): referee panel -- independent COBOL/CICS parsers scored per fact channel (#4377) - #4384

test(cobol): referee panel -- independent COBOL/CICS parsers scored per fact channel (#4377)#4384
squid-protocol merged 2 commits into
mainfrom
feat/4377-referee-panel

squid-protocol commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

squid-protocol commented Oct 5, 2026

What this adds (test tooling only)

Referee panel scorecard (#4377)

Parse success and speed (the key-covered members each source reads: programs and / or keyed copybooks)

Each source vs the answer keys (ground truth), all corpora pooled

Agreement with the engine (|A ∩ B| / |A ∪ B| on files both parsed)

How to read it

Adjudicated disagreements (at least 3 sampled per pattern, against the source)

Draft issues (not filed)

Draft A (engine)

Draft B (engine, minor)

Recommendation for the next phase

Uh oh!

github-actions Bot commented Oct 5, 2026

🐦‍⬛ Muninn Security Scan

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants