Repository navigation
Refractor: DD lineage, dynamic CALLs and schemas from the engine's fact channels (#3348) - #3768
Merged
Merged
Conversation
…ct channels (#3348) With a master DB, process_payload now reads a program's DD lineage and dynamic CALLs (dataset_data + call_site_data) and its generated schemas (record_data) through the new engine_sourcing.py instead of re-parsing the source; metadata.ir_sources records the source per field, and the forge readers are the fallback without a DB or with a DB from before a channel. Dead code stays on the forge: its reachability analysis names the dead paragraphs and the engine's facts in them are dropped by line. For that, each COBOL dataset row now records its OPEN sites (dataset_data.open_sites, [mode, line] JSON; NULL on JCL rows and older DBs) -- written, healed onto older tables, restored on delta scans and read by GalaxyIR. EngineFile.records_read tells "no items" from "no record_data table". cobol_schema_forge splits into its reader and render_schemas, shared by both sources. The refraction differential compares both as data now (`lineage`, `schema_column`, unexplained on any delta) instead of listing the forge's side as stated_absence. Measured on the six full pinned corpora (123 COBOL programs): identical on every program (92 DD edges, 4 dynamic CALLs, 5,475 schema columns); 0 unexplained on every excerpt and full corpus. Baselines re-blessed (by_cause only; carddemo full 15 -> 0 from earlier merges, noted). Snapshots re-blessed: the only change is the ir_sources block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g
Contributor
squid-protocol
added a commit
that referenced
this pull request
Sep 26, 2026
…nerations, reuse them (#3777) The pull-request check (java_target_matrix.py --same-as) now: - compares only what the build reads (src/, pom.xml, the Gradle files). On #3768 every difference was a porting ticket (ai_agent_jobs/*.json), yet four corpora set up the JDKs and compiled every config twice; - runs its four generations (head/base x plain/scan) side by side; - caches the base's generation per base commit (--base-cache), so a later push to the same PR generates only the head; - and, when a build input did change, the build steps compile the projects the check already generated (--generated) instead of refactoring -- and engine-scanning -- the corpus a third time. Pushes, the weekly run and manual runs still build from scratch. dsf (the slowest check, 416s on CI): 170s -> 104s cold, 85s with the base cached, locally; the remainder is the head's own engine scan. Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 26, 2026
squid-protocol
pushed a commit
that referenced
this pull request
Sep 26, 2026
…ive paths) The rehydrated ram_cache is keyed by native paths (src\LEDGER.cbl on Windows), so indexing it with "src/LEDGER.cbl" failed the Windows matrix since #3768. Test-only: look the key up via a _cached(cache, rel) helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AbvXUJpva5XxxbssoKG9Jt
squid-protocol
added a commit
that referenced
this pull request
Sep 26, 2026
#3779, #3780) (#3795) * Call-graph skills: language onboarding, miss sweep, Haiku triage scout (#3779, #3780) - .claude/skills/callgraph-language-onboarding: the checklist for adding a language's compiler reference (tool/pin traps, corpus choice, adapter naming rules, registry, fixture test, sanity checks, baseline/CI/hook, docs). - .claude/skills/callgraph-miss-sweep: triage -> known-pattern table -> source verification -> explain_call root cause -> fix -> net-change rule -> ship. - .claude/agents/callgraph-triage-scout (haiku): mechanical triage/split/explain runs and issue drafts; pastes tool output verbatim, never judges. - callgraph_triage.py: --split BUCKET classifies a bucket's call sites by shape and receiver deterministically (a Haiku trial misclassified them by eye); --json samples now carry their source excerpt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AbvXUJpva5XxxbssoKG9Jt * Fix Windows KeyError in refractor engine-channel tests (ram_cache native paths) The rehydrated ram_cache is keyed by native paths (src\LEDGER.cbl on Windows), so indexing it with "src/LEDGER.cbl" failed the Windows matrix since #3768. Test-only: look the key up via a _cached(cache, rel) helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AbvXUJpva5XxxbssoKG9Jt --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3348. This is the last concrete piece of epic #3122.
What changes
With a master DB, the refractor no longer re-parses what the engine already extracted.
process_payloadtakes three things from the engine (via the newgitgalaxy/tools/cobol_to_cobol/engine_sourcing.py):analysis.lineage(DD inputs / outputs)extract_lineagedataset_dataanalysis.lineage.unresolved_callsextract_lineagecall_site_datageneration.schemasforge_schemasrecord_datametadata.ir_sourcesrecords whethergalaxy_dbor theforgesupplied each field.usage_statusis never used for masking.dataset_data.open_sites, a JSON list of[mode, line]. It is written, healed onto older tables, restored on delta scans and read byGalaxyIR. It is NULL on JCL rows and on older DBs.cobol_schema_forgeis split into its reader and a sharedrender_schemas, so both sources render identically.Acceptance
lineageandschema_columnare compared data, and any disagreement isunexplained. Previously the forge's side was listed asstated_absence.--cistays green at 0 unexplained on every excerpt, and--corpusis at 0 on every full corpus.test_refractor_engine_channels.py: engine-sourced; forge fallback without a DB; and a DB from before the channels (noopen_sitescolumn, norecord_data).galaxy_ir.pySCOPE comment, theprocess_payloaddocstring anddocs/refraction_engine_differential.md(new update section) now match what is consumed.Re-blessed
by_causecounts change, since stated absences drop now that lineage is compared. The full-corpus bless also picks up drift from earlier merges. The notable one is CardDemo full, from 15 unexplained to 0: those were copybook/PROGRAM-ID deltas this PR doesn't touch, closed by Forge copybook fallback: follow EXEC SQL INCLUDE and COPY … IN <library> #3414 and the CMQ* copybooks now classified as system copybooks. The note says so.ir_sourcesblock. Snapshots run without a DB, so they showforge.Noted, not changed
An
OCCURS ... DEPENDING ONon a group item gets no "use JSONB" note, from either source, because the flat schema lists only elementary items. This is pre-existing and identical in both sources.🤖 Generated with Claude Code
https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g