Repository navigation
Refraction pipeline: consume the engine's master DB instead of re-parsing COBOL #3120
Description
Found by a design review of the COBOL→Java modernization suite (gitgalaxy/tools/cobol_to_java/, gitgalaxy/tools/cobol_to_cobol/, gitgalaxy/cobol_refractor_controller.py, gitgalaxy/cobol_to_java_controller.py) against the engine's mainframe language coverage.
Part of #3122.
The modernization pipeline and the analysis engine share nothing
Grepping galaxy_master|galaxyscope|_galaxy_audit|LANGUAGE_DEFINITIONS|signal_processor across both tool suites and both controllers returns 16 hits, every one of them a # galaxyscope:ignore pragma comment — no imports, no artifact reads, zero real references. The two halves of the product are fully decoupled.
So the pipeline re-implements COBOL parsing
analyze_cobol_intent (gitgalaxy/tools/cobol_to_cobol/cobol_jcl_forge.py:24-80) carries its own:
- fixed-format margin normalizer — skips lines where
line[6] in ("*", "/"), slicesline[7:]; PROGRAM-IDregex;SELECT ... ASSIGNfile-assignment parser;EXEC CICS/EXEC SQLdetection viare.findall.
It is COBOL-only, and it is a second parser for a language the engine already parses under audit.
The engine already produces the IR the pipeline wants
A single galaxyscope run emits <repo>_galaxy_master.db with repo_data, folder_data, file_data, class_data, function_data, edge_data, excluded_artifacts. The relevant columns (all verified in gitgalaxy/recorders/record_keeper.py):
| table | columns |
|---|---|
function_data |
func_name, complexity, loc, start_line, args, usage_status, func_archetype, docstring, calls_out_to, is_public, is_documented |
file_data |
language, file_path, total_loc, coding_loc, import_count, pagerank_score, normalized_blast_radius, directory_group |
edge_data |
src_file_id, dst_file_id, edge_kind, import_statements, entity_imports |
There is also <repo>_galaxy_graph.sqlite with artifacts, functions, outbound_dependencies, inbound_dependencies, dna_hits.
Concrete proof of the multi-language gap
Scanning IBM's zopeneditor-sample (a real COBOL/PL-I/JCL/HLASM/REXX CICS sample repo, already in training_repos/cobol_corpus/) produced 107 extracted units across five mainframe languages in one DB: cobol 70, jcl 25, pli 7, rexx 4, hlasm 1.
The refraction pipeline can currently act on the cobol 70 only, and only by re-parsing them itself.
The ask
Make cobol_refractor_controller / cobol_to_java_controller accept a <repo>_galaxy_master.db (or the graph sqlite) as their IR source, so that:
- every improvement to the
cobolprofile reaches the product automatically — today engine hardening and product capability are disconnected; usage_statusreplaces the re-derived dead-code x-ray. The dead-code determination thatx_ray_dead_code(gitgalaxy/tools/cobol_to_cobol/cobol_graveyard_finder.py, imported atcobol_refractor_controller.py:25, called at:189) re-derives is already a committed column. Verified live on the zopeneditor scan:usage_statusis populated, flagging 5 of the 70 cobol units;edge_datareplaces the hand-rolled lineage extraction;- the pipeline becomes language-agnostic by construction — the same query returns PL/I, JCL, REXX and HLASM units. This is the prerequisite for every phase-2 item in the epic.
Transition risk — do not skip this
The two parsers will disagree somewhere. Before the switch, do a differential run over a real corpus comparing old-parser vs engine-DB extraction, and explain the deltas rather than assuming they are improvements. Both cics-banking-sample-application-cbsa and zopeneditor-sample are already in training_repos/cobol_corpus/ and are the natural subjects.
Secondary motivation: discipline asymmetry
The forge regexes are held to none of the engine's standards. Example, from the same analyze_cobol_intent:
cics_matches = re.findall(r"EXEC\s+CICS.*?END-EXEC\.", monolith_code, re.IGNORECASE)That is an unbounded lazy quantifier run over a whole-file-joined single string (monolith_code = " ".join(clean_lines)) — the shape how_to_add_a_language.md Rule 5 ("Absolute ReDoS Immunity: bound all wildcards") and Rule 14 (adjacent quantifiers with overlapping classes) explicitly forbid for engine rules. Forge patterns get no ReDoS audit, no strict suite and no corpus validation, while the cobol profile's equivalents get all three.
Consuming the engine's DB means one audited parser instead of two, one of which is unaudited.
Activity
squid-protocol commentedon Sep 19, 2026
Differential done (the "transition risk — do not skip this" step), with PR #3207 implementing the part it supports. The full explained report is
docs/refraction_engine_differential.mdin that PR.Two of this issue's four premises don't hold on main yet:
usage_statuscan't replace the dead-code x-ray. It is a same-file by-name test.000-MAIN,PREMIERE SECTION), is case-sensitive, and letsX-EXITcount as a reference toX(cobol usage_status: entry paragraph flagged, case-sensitive matching, and NAME-EXIT counts as a reference to NAME #3198).END-IF./GOBACK.; on CBSA, 402 of its 623 are terminators, inlined foreign programs, or SECTION fall-through (Refraction graveyard finder: scope terminators as paragraphs, programs inlined as copybooks, same-directory-only COPY, no SECTION model #3203).edge_datacan't replace lineage. It holds COPY/INCLUDE edges only: no SELECT/ASSIGN, no OPEN modes, no CALL, noEXEC PGM=(Mainframe call graph: no edges for COBOL CALL, CICS LINK/XCTL or JCL EXEC PGM=, and unresolved CALL targets are recorded nowhere #3200, Extract COBOL SELECT/ASSIGN and OPEN modes (and JCL DD bindings) by name, so dataset lineage can come from the engine #3201). The old DAG architect has never reported an output on either corpus (Refraction DAG architect: multi-mode OPEN assigns the first mode to every file, so no program ever has an output (all JCL DDs DISP=SHR) #3204).What does hold, and what #3207 switches on (opt-in
--galaxy-db/--scan):Left for this issue: move dead code to the DB once #3198 lands, and lineage once #3200 + #3201 land. The differential harness (
tests/tools/refraction_differential.py) is the check to re-run at each switch.fix(#3120): key galaxy IR paths by POSIX form (Windows DBs store back…
feat(#3120): refraction pipeline reads the engine master DB (opt-in I…
feat(#3210): hand-verified COBOL answer key for the #3120 corpora, sc…
fix(cobol): sentence-anchored paragraph headers (#3197) + census iden…
feat(mainframe): named call graph and dataset boundary (#3200, #3201) (…
Refraction differential: classify every delta by cause and gate on an…