Skip to content

Refraction pipeline: consume the engine's master DB instead of re-parsing COBOL #3120

Closed

Description

@squid-protocol

Found by a design review of the COBOL→Java modernization suite (gitgalaxy/tools/cobol_to_java/, gitgalaxy/tools/cobol_to_cobol/, gitgalaxy/cobol_refractor_controller.py, gitgalaxy/cobol_to_java_controller.py) against the engine's mainframe language coverage.

Part of #3122.

The modernization pipeline and the analysis engine share nothing

Grepping galaxy_master|galaxyscope|_galaxy_audit|LANGUAGE_DEFINITIONS|signal_processor across both tool suites and both controllers returns 16 hits, every one of them a # galaxyscope:ignore pragma comment — no imports, no artifact reads, zero real references. The two halves of the product are fully decoupled.

So the pipeline re-implements COBOL parsing

analyze_cobol_intent (gitgalaxy/tools/cobol_to_cobol/cobol_jcl_forge.py:24-80) carries its own:

  • fixed-format margin normalizer — skips lines where line[6] in ("*", "/"), slices line[7:];
  • PROGRAM-ID regex;
  • SELECT ... ASSIGN file-assignment parser;
  • EXEC CICS / EXEC SQL detection via re.findall.

It is COBOL-only, and it is a second parser for a language the engine already parses under audit.

The engine already produces the IR the pipeline wants

A single galaxyscope run emits <repo>_galaxy_master.db with repo_data, folder_data, file_data, class_data, function_data, edge_data, excluded_artifacts. The relevant columns (all verified in gitgalaxy/recorders/record_keeper.py):

table columns
function_data func_name, complexity, loc, start_line, args, usage_status, func_archetype, docstring, calls_out_to, is_public, is_documented
file_data language, file_path, total_loc, coding_loc, import_count, pagerank_score, normalized_blast_radius, directory_group
edge_data src_file_id, dst_file_id, edge_kind, import_statements, entity_imports

There is also <repo>_galaxy_graph.sqlite with artifacts, functions, outbound_dependencies, inbound_dependencies, dna_hits.

Concrete proof of the multi-language gap

Scanning IBM's zopeneditor-sample (a real COBOL/PL-I/JCL/HLASM/REXX CICS sample repo, already in training_repos/cobol_corpus/) produced 107 extracted units across five mainframe languages in one DB: cobol 70, jcl 25, pli 7, rexx 4, hlasm 1.

The refraction pipeline can currently act on the cobol 70 only, and only by re-parsing them itself.

The ask

Make cobol_refractor_controller / cobol_to_java_controller accept a <repo>_galaxy_master.db (or the graph sqlite) as their IR source, so that:

  1. every improvement to the cobol profile reaches the product automatically — today engine hardening and product capability are disconnected;
  2. usage_status replaces the re-derived dead-code x-ray. The dead-code determination that x_ray_dead_code (gitgalaxy/tools/cobol_to_cobol/cobol_graveyard_finder.py, imported at cobol_refractor_controller.py:25, called at :189) re-derives is already a committed column. Verified live on the zopeneditor scan: usage_status is populated, flagging 5 of the 70 cobol units;
  3. edge_data replaces the hand-rolled lineage extraction;
  4. the pipeline becomes language-agnostic by construction — the same query returns PL/I, JCL, REXX and HLASM units. This is the prerequisite for every phase-2 item in the epic.

Transition risk — do not skip this

The two parsers will disagree somewhere. Before the switch, do a differential run over a real corpus comparing old-parser vs engine-DB extraction, and explain the deltas rather than assuming they are improvements. Both cics-banking-sample-application-cbsa and zopeneditor-sample are already in training_repos/cobol_corpus/ and are the natural subjects.

Secondary motivation: discipline asymmetry

The forge regexes are held to none of the engine's standards. Example, from the same analyze_cobol_intent:

cics_matches = re.findall(r"EXEC\s+CICS.*?END-EXEC\.", monolith_code, re.IGNORECASE)

That is an unbounded lazy quantifier run over a whole-file-joined single string (monolith_code = " ".join(clean_lines)) — the shape how_to_add_a_language.md Rule 5 ("Absolute ReDoS Immunity: bound all wildcards") and Rule 14 (adjacent quantifiers with overlapping classes) explicitly forbid for engine rules. Forge patterns get no ReDoS audit, no strict suite and no corpus validation, while the cobol profile's equivalents get all three.

Consuming the engine's DB means one audited parser instead of two, one of which is unaudited.

Activity

  1. added
    enhancementNew feature, sensor, or structural signature
    core-engineModifications to the central physics and parsing engine
    legacy-modernizationCOBOL refractor, dead-code extraction, and JCL forging
    on Sep 17, 2026
  2. squid-protocol commented on Sep 19, 2026

    @squid-protocol
    OwnerAuthor

    Differential done (the "transition risk — do not skip this" step), with PR #3207 implementing the part it supports. The full explained report is docs/refraction_engine_differential.md in that PR.

    Two of this issue's four premises don't hold on main yet:

    What does hold, and what #3207 switches on (opt-in --galaxy-db / --scan):

    Left for this issue: move dead code to the DB once #3198 lands, and lineage once #3200 + #3201 land. The differential harness (tests/tools/refraction_differential.py) is the check to re-run at each switch.

  3. added 2 commits that reference this issue on Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    core-engineModifications to the central physics and parsing engineenhancementNew feature, sensor, or structural signaturelegacy-modernizationCOBOL refractor, dead-code extraction, and JCL forging

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions