Mainframe fact channel: IBM MQ calls -- queues, direction, handles (#3447) by squid-protocol · Pull Request #3461 · squid-protocol/gitgalaxy

squid-protocol · 2026-09-24T04:12:57Z

Closes #3447. Part of #3445. Stacked on #3460, which sits on #3459 and #3458. GitHub retargets each PR as the one below it merges.

What

New fact channel mq_call_data (core/mq_calls.py). It records one row per MQI call, with:

the verb;
the direction (from the MQOO-* open options; PUT is put, GET is get);
the queue;
the handle;
the option words in effect.

How the queue is read. MQ names its queue through a descriptor, never through a call operand.

An MQOPEN or MQPUT1 uses the operand last MOVEd to its descriptor's OBJECTNAME (qualified with OF <od> or not).
That operand is read as a literal, its VALUE, or the literals MOVEd to it, following name-to-name MOVEs.
Two runtime sources are named, never guessed:
- trigger: MQTM-QNAME, the queue whose trigger started the transaction;
- reply_to: MQMD-REPLYTOQ, the requester's reply-to queue.

Handles. A PUT, GET or CLOSE is matched to its MQOPEN by handle. That's either the handle the call passes, or the one MOVEd into its handle operand just before it. Copies of an open's handle count too: CardDemo opens every queue into a shared MQ-HOBJ, then runs MOVE MQ-HOBJ TO INPUT-QUEUE-HANDLE.

Joins:

GalaxyIR.mq_queues() lists every program's queue endpoints, with runtime queues named.
GalaxyIR.mq_flows() pairs producers with consumers through a named queue.

CardDemo

program	reads	writes
COACCT01	`<trigger>`	`CARD.DEMO.REPLY.ACCT`, `CARD.DEMO.ERROR`
CODATE01	`<trigger>`	`CARD.DEMO.REPLY.DATE`, `CARD.DEMO.ERROR`
COPAUA0C	`<trigger>`	`MQPUT1` to `<reply_to>`

Every PUT, GET and CLOSE is matched to its own MQOPEN. These are request/reply services whose requesters live outside the repository, so there's no in-repo producer→consumer pair. mq_flows is empty, which is correct for this corpus; a unit test covers the pairing.

Ground truth

Independent reader: add-mq, a token walk with its own MOVE, COMPUTE and handle tracking.
New scored field: MQ calls, drafted, with sign-off flag mq_validated.
Result: the engine matches the key exactly: CardDemo 22/22. CBSA and zopeneditor make no MQ calls.
The ledger earned its keep here. On the real scan, the first cut scored 4/22. In sequence-numbered source (COACCT01, CODATE01), the columns 73–80 tag was ending each CALL's operand list. The fix blanks sequence fields first, and a regression test covers it.

Checks

Golden masters: unchanged, since the crucible corpus has no MQ program.
Tests: full suite passes locally (11,167 passed). The ruff, mypy and dead-key audits show no new findings.

🤖 Generated with Claude Code

https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g

…3447) CALL 'MQOPEN' / 'MQPUT' / 'MQGET' reached call_site_data as external calls only; nothing said which program puts to or gets from which queue. New channel mq_call_data (core/mq_calls.py): one row per MQI call with the verb, direction (MQOO-* open options; PUT put, GET get), the queue, the handle and the option words in effect. - The queue of an MQOPEN / MQPUT1 is the operand last MOVEd to its descriptor's OBJECTNAME (qualified `OF <od>` or not), read as a literal / VALUE / MOVEd literals, following name-to-name MOVEs (depth 3). Two runtime sources are named, never guessed: `trigger` (MQTM-QNAME, the queue whose trigger started the transaction) and `reply_to` (MQMD-REPLYTOQ). - A PUT / GET / CLOSE is matched to its MQOPEN by handle: the one it passes, or the one MOVEd into its handle operand just before it, with the copies an open's handle gets before the next MQ call (`MOVE MQ-HOBJ TO INPUT-QUEUE-HANDLE`) -- so CardDemo's shared MQ-HOBJ still resolves. - Fixed-format sequence fields are blanked first: in numbered source the cols 73-80 tag used to end a CALL's operand list. The ground-truth ledger caught this on the real scan (COACCT01/CODATE01 4/22 -> 22/22). Spine per how_to_add_a_fact_channel.md: extract_boundary (cobol), payload, record_keeper table + indexes, byte-identical delta restore, audit block, LLM per-file queue line; galaxy_ir EngineMqCall / EngineFile.mq_calls, GalaxyIR.mq_queues (per-program endpoints, runtime queues named) and mq_flows (producer -> consumer through a named queue). Ground truth: an independent token-walk reader (add-mq) with its own MOVE, COMPUTE and handle tracking; new scored field "MQ calls" (mq_validated, drafted). Engine matches the key exactly: CardDemo 22/22; CBSA and zopeneditor make no MQ calls. CardDemo's MQ programs are request/reply services whose requesters are outside the repo, so no in-repo producer->consumer pair exists. Golden masters unchanged (no MQ program in the crucible corpus). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g

github-actions · 2026-09-24T04:17:34Z

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

…3448) (#3460) * Mainframe fact channel: embedded SQL table access (#3446) sql_table_data (#3344) held DB2 table SHAPES; nothing said which programs use which table, or how. New channel sql_statement_data: every embedded SQL statement (EXEC SQL ... END-EXEC in COBOL, ... ; in PL/I), one row per (statement, table), with the verb, access (read / insert / update / delete / merge / lock), cursor and host variables. A statement naming no table (OPEN / FETCH / CLOSE, COMMIT, CALL) keeps one row, so the cursor lifecycle and unit-of-work points survive. INCLUDE, DECLARE ... TABLE and WHENEVER are not statements; dynamic SQL names no table. Spine per how_to_add_a_fact_channel.md: - core/db2_sql_statements.py: bounded extractor reusing db2_declare_table's sequence-field, literal and comment helpers. - Wired through extract_boundary, galaxyscope, record_keeper (+ indexes), state_rehydrator (byte-identical delta restore) and the audit block and LLM brief. - galaxy_ir: EngineSqlStatement, EngineFile.sql_statements, and the join GalaxyIR.sql_table_access(), a program x table matrix where OPEN / FETCH reach the cursor's DECLAREd SELECT. Ground truth: an independent token-walk reader in cobol_answer_key.py (add-sql-access), a new "DB2 table access" field (TRUTH_FLAGS sql_access_validated, drafted), and every key drafted. The engine matches the independent reader exactly: CardDemo 13/13, CBSA 24/24 (zopeneditor has no SQL). Every non-comment EXEC SQL in the corpora is accounted for. Golden masters re-blessed: 23 differences, all new `SQL Statements` blocks on COBOL files, no other movement. keyword-rosetta identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g * test_galaxy_ir: drop redundant local shutil/sqlite3 imports (CodeQL) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g * Mainframe fact channel: CICS task control and the async task graph (#3449) Only RUN TRANSID reached call_site_data, as a routing verb whose target never resolved; the async API's join points, START/RETRIEVE data passing and DELAY/POST/WAIT/ENQ were not modelled, so CBSA's credit-agency fan-out/fan-in read as one unconnected child start. New channel cics_task_data: one row per CICS task-control command -- RUN, START, START ATTACH, FETCH CHILD / FETCH ANY, FREE CHILD, RETRIEVE, CANCEL, DELAY, POST, WAIT EVENT / WAIT EXTERNAL / WAITCICS, ENQ, DEQ -- with the TRANSID or RESOURCE target, the CHANNEL, the CHILD / ANY / REQID token, the FROM/INTO record, the timing and the remaining options. Target resolution adds one reading to literal / VALUE / MOVE: a transaction id BUILT by STRING becomes an fnmatch pattern, each source contributing its literal text or its PIC width (`[0-9]` per digit, `?` per character). CRECUST's `STRING 'OCR' ..., WS-CC-CNT (PIC 9) ... INTO WS-RUN-TRANSID` is `OCR[0-9]`: it reaches OCR1..OCR5 and deliberately not OCRA, another transaction in the same deck. A STRING also assigns the name, so it joins any MOVEd literals as candidates rather than losing to a sole MOVE. Spine per how_to_add_a_fact_channel.md: - core/cics_tasks.py: bounded extractor reusing cics_resources' block and option walker; STRING reader bounded by _STRING_LIMIT and the sentence end. - extract_boundary (cobol + pli), galaxyscope payload, record_keeper table + indexes, state_rehydrator byte-identical delta restore, audit block, LLM brief summary and per-file line. - galaxy_ir: EngineCicsTask, EngineFile.cics_tasks, and GalaxyIR.async_tasks(): each RUN/START joined to the CSD transactions and programs it can reach, the containers the parent PUTs on the passed channel, the FETCH ANY / FETCH CHILD / FREE CHILD that collect it (by token), and a START's child RETRIEVEs. CBSA: CRECUST RUN -> OCR1..OCR5 -> CRDTAGY1..5, containers CIPA..CIPI on CIPCREDCHANN, collected by FETCH ANY at line 639. Ground truth: an independent reader in cobol_answer_key.py (add-cics-tasks) with its own STRING/PIC reading and CSD expansion; two new scored fields, "CICS task control" and "async children" (TRUTH_FLAGS cics_tasks_validated, drafted). The engine matches the key exactly: CBSA 24/24 commands and 5/5 children, CardDemo 3/3 (its MQ-triggered RETRIEVEs), zopeneditor none. No START TRANSID exists in any pinned corpus; START is covered by unit tests. Golden masters re-blessed: the only change is a new cics_tasks.json in each fixture (CBSA + genapp COBOL files); no existing golden file moved. dead_key_audit: "CHANNEL" allowlisted -- it is a parsed EXEC CICS option word. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g * Mainframe fact channel: job submission through the internal reader (#3448) The engine recorded both halves of an online->batch submission -- the CICS WRITEQ TD (cics_resource_data) and the extrapartition TDQUEUE (csd_resource_data) -- but not the edge "transaction X submits job Y", and nothing of the batch case, a step whose DD is SYSOUT=(A,INTRDR). New channel job_submit_data (core/job_submits.py): - COBOL: the JCL cards a program holds as literals -- `//NAME JOB` (the job name) and `//STEP EXEC PROC=|PGM=|P` -- kept only when a JOB card is among them, so a DISPLAY '//...' is never JCL. One bounded literal regex, shielded. - JCL: every DD routed to the internal reader, with the step's SYSUT1 DSN (what IEBGENER copies there), read from the existing _jcl_statements parse. galaxy_ir: EngineJobSubmit, EngineFile.job_submits, and GalaxyIR.job_submissions(): - online (`tdq`): WRITEQ TD to an extrapartition TDQUEUE, when some JCL routes its DDNAME to SYSOUT=(x,INTRDR) (`region_jcl`) or the writer embeds a job card (`job_card`); with the transactions that enter the writer, the job names and each EXEC target resolved to its JCL member (a PROC prefers the procedure member: TRANREPT.prc over TRANREPT.jcl). - batch (`intrdr_dd`): the SYSUT1 library member resolved to its JCL file. CardDemo: CR00 -> CORPT00C -> TDQ JOBS (INREADER) -> job TRNRPT00 -> PROC TRANREPT = app/proc/TRANREPT.prc; INTRDRJ1 -> INTRDRJ2. Ground truth: an independent join in cobol_answer_key.py (add-job-submissions) with its own line-by-line JCL read, job-card scan and CSD / WRITEQ readers; new scored field "job submissions" (TRUTH_FLAGS submissions_validated, drafted). Engine matches the key exactly: CardDemo 3/3; CBSA and zopeneditor submit nothing. Golden masters re-blessed: one new Job Submission block (CORPT00C), nothing else moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g * Mainframe fact channel: CICS task control and the async task graph (#3449) (#3459) Only RUN TRANSID reached call_site_data, as a routing verb whose target never resolved; the async API's join points, START/RETRIEVE data passing and DELAY/POST/WAIT/ENQ were not modelled, so CBSA's credit-agency fan-out/fan-in read as one unconnected child start. New channel cics_task_data: one row per CICS task-control command -- RUN, START, START ATTACH, FETCH CHILD / FETCH ANY, FREE CHILD, RETRIEVE, CANCEL, DELAY, POST, WAIT EVENT / WAIT EXTERNAL / WAITCICS, ENQ, DEQ -- with the TRANSID or RESOURCE target, the CHANNEL, the CHILD / ANY / REQID token, the FROM/INTO record, the timing and the remaining options. Target resolution adds one reading to literal / VALUE / MOVE: a transaction id BUILT by STRING becomes an fnmatch pattern, each source contributing its literal text or its PIC width (`[0-9]` per digit, `?` per character). CRECUST's `STRING 'OCR' ..., WS-CC-CNT (PIC 9) ... INTO WS-RUN-TRANSID` is `OCR[0-9]`: it reaches OCR1..OCR5 and deliberately not OCRA, another transaction in the same deck. A STRING also assigns the name, so it joins any MOVEd literals as candidates rather than losing to a sole MOVE. Spine per how_to_add_a_fact_channel.md: - core/cics_tasks.py: bounded extractor reusing cics_resources' block and option walker; STRING reader bounded by _STRING_LIMIT and the sentence end. - extract_boundary (cobol + pli), galaxyscope payload, record_keeper table + indexes, state_rehydrator byte-identical delta restore, audit block, LLM brief summary and per-file line. - galaxy_ir: EngineCicsTask, EngineFile.cics_tasks, and GalaxyIR.async_tasks(): each RUN/START joined to the CSD transactions and programs it can reach, the containers the parent PUTs on the passed channel, the FETCH ANY / FETCH CHILD / FREE CHILD that collect it (by token), and a START's child RETRIEVEs. CBSA: CRECUST RUN -> OCR1..OCR5 -> CRDTAGY1..5, containers CIPA..CIPI on CIPCREDCHANN, collected by FETCH ANY at line 639. Ground truth: an independent reader in cobol_answer_key.py (add-cics-tasks) with its own STRING/PIC reading and CSD expansion; two new scored fields, "CICS task control" and "async children" (TRUTH_FLAGS cics_tasks_validated, drafted). The engine matches the key exactly: CBSA 24/24 commands and 5/5 children, CardDemo 3/3 (its MQ-triggered RETRIEVEs), zopeneditor none. No START TRANSID exists in any pinned corpus; START is covered by unit tests. Golden masters re-blessed: the only change is a new cics_tasks.json in each fixture (CBSA + genapp COBOL files); no existing golden file moved. dead_key_audit: "CHANNEL" allowlisted -- it is a parsed EXEC CICS option word. Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Mainframe fact channel: IBM MQ calls -- queues, direction, handles (#3447) (#3461) CALL 'MQOPEN' / 'MQPUT' / 'MQGET' reached call_site_data as external calls only; nothing said which program puts to or gets from which queue. New channel mq_call_data (core/mq_calls.py): one row per MQI call with the verb, direction (MQOO-* open options; PUT put, GET get), the queue, the handle and the option words in effect. - The queue of an MQOPEN / MQPUT1 is the operand last MOVEd to its descriptor's OBJECTNAME (qualified `OF <od>` or not), read as a literal / VALUE / MOVEd literals, following name-to-name MOVEs (depth 3). Two runtime sources are named, never guessed: `trigger` (MQTM-QNAME, the queue whose trigger started the transaction) and `reply_to` (MQMD-REPLYTOQ). - A PUT / GET / CLOSE is matched to its MQOPEN by handle: the one it passes, or the one MOVEd into its handle operand just before it, with the copies an open's handle gets before the next MQ call (`MOVE MQ-HOBJ TO INPUT-QUEUE-HANDLE`) -- so CardDemo's shared MQ-HOBJ still resolves. - Fixed-format sequence fields are blanked first: in numbered source the cols 73-80 tag used to end a CALL's operand list. The ground-truth ledger caught this on the real scan (COACCT01/CODATE01 4/22 -> 22/22). Spine per how_to_add_a_fact_channel.md: extract_boundary (cobol), payload, record_keeper table + indexes, byte-identical delta restore, audit block, LLM per-file queue line; galaxy_ir EngineMqCall / EngineFile.mq_calls, GalaxyIR.mq_queues (per-program endpoints, runtime queues named) and mq_flows (producer -> consumer through a named queue). Ground truth: an independent token-walk reader (add-mq) with its own MOVE, COMPUTE and handle tracking; new scored field "MQ calls" (mq_validated, drafted). Engine matches the key exactly: CardDemo 22/22; CBSA and zopeneditor make no MQ calls. CardDemo's MQ programs are request/reply services whose requesters are outside the repo, so no in-repo producer->consumer pair exists. Golden masters unchanged (no MQ program in the crucible corpus). Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * db2_sql_statements: build long regexes from named pieces (X-Ray entropy) The X-Ray entropy scan flagged three long regex literals. Build them from named parts (_S, _ALIAS_STOP, _FETCH_ORIENTATION, _HOST_NAME) instead; every compiled pattern is byte-identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g * job_submits: build the JCL card regex from a named piece (X-Ray entropy) Same fix as db2_sql_statements in #3458: the compiled pattern is byte-identical; the literal no longer reads as a dense string. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

squid-protocol merged commit 4b51bfc into feat/job-submits-3448 Sep 24, 2026
2 checks passed

squid-protocol deleted the feat/mq-channel-3447 branch September 24, 2026 08:53

squid-protocol mentioned this pull request Sep 24, 2026

Mainframe fact channel: MQ messaging: queues, put/get direction, producer/consumer graph #3447

Closed

5 tasks

Navigation Menu

Repository navigation

Mainframe fact channel: IBM MQ calls -- queues, direction, handles (#3447) - #3461

Mainframe fact channel: IBM MQ calls -- queues, direction, handles (#3447)#3461
squid-protocol merged 1 commit into
feat/job-submits-3448from
feat/mq-channel-3447

squid-protocol commented Sep 24, 2026

Uh oh!

github-actions Bot commented Sep 24, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

squid-protocol commented Sep 24, 2026

What

CardDemo

Ground truth

Checks

Uh oh!

github-actions Bot commented Sep 24, 2026

🐦‍⬛ Muninn Security Scan

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant