Repository navigation
COBOL modernization answer key: hand-verified per-program truth for the #3120 corpora, scored against forge and engine #3210
Description
Part of #3122. Tooling item 1 from the #3120 retrospective.
Problem
The #3120 differential (docs/refraction_engine_differential.md) spent most of its time establishing ground truth by hand:
- which paragraphs are really dead (PERFORM / GO TO / SECTION fall-through)
- which copybooks a COPY really means
- which files a program really opens for output
- whether a block is really
EXEC CICS
Neither the refraction tools nor the engine has an answer key for real mainframe source. The forge tools' tests are all synthetic, which is how "every dead paragraph on zopeneditor is END-IF." (#3203) and "no program ever has an output" (#3204) shipped. Each of #3197–#3206 would currently need that investigation repeated.
Proposal
A hand-verified, per-program answer key for the two #3120 corpora at pinned commits:
- IBM/zopeneditor-sample
8f98353 - cicsdev/cics-banking-sample-application-cbsa
4173345(branchJuly2024Refresh)
For every COBOL program it records:
| field | content |
|---|---|
program_id |
PROGRAM-ID |
units |
real paragraph and section headers (name, kind, line) |
dead |
units unreachable from the entry, with the reason each is dead |
copybooks |
each COPY / EXEC SQL INCLUDE → the repo file it means, or null plus why (system-supplied, BMS-generated) |
files |
SELECT internal name → DD name, plus the OPEN modes actually used |
calls |
CALL targets, static or dynamic, and what they resolve to |
cics, sql |
presence |
A scorer reports per-field agreement for both the forge tools and the engine DB. The same key then measures each #3197–#3206 fix, and the remaining #3120 switches.
Authority
A draft helper computes candidate values with its own fixed-format-aware reading: Area A headers, PERFORM … THRU, GO TO, SECTION fall-through, terminal statements. The draft is not the key. Each dead verdict, each value where the draft, the forge and the engine disagree, and a spot sample of agreements are checked against the source, and the result is recorded as validated with a note. This follows the same status discipline as the tri-comparison ledger. The helper is a test tool, not a third product parser.
Done when
- The key covers both corpora and is committed under
tests/cobol_mainframe/answer_key/. - The scorer runs against a checkout plus a scan.
- The current forge and engine scores are recorded in the issue.
Activity
fix(#3210): commit the answer key JSONs (the repo-wide *.json ignore …
fix(#3199): resolve an ambiguous COPY target instead of dropping the …
Refraction differential: classify every delta by cause and gate on an…