Repository navigation
Ground truth: blind census of the fact-channel key sections (#3445) - #3467
Merged
Merged
Conversation
The six answer-key sections drafted by epic #3445 (SQL table access, CICS task control + async children, MQ calls, units of work and handlers, job submissions, TD trigger starts) matched the engine exactly but were only the drafting model's reading. They have now been read a second time, blind. tests/tools/cross_verify_sections.py: the cross_verify census for per-file sections. It asks EVERY COBOL file of a corpus (files with no key entry too, so recall is checked, not just precision), packed into batches by fact count, with the corpus-wide questions (job submissions, TD triggers) in batch 1. The brief states each channel's contract and never the key's answers; reviewer facts are put into the same canonical strings as the key's and compared per file. grade / rulings / sign as in cross_verify.py; signing sets the section's *_validated flag with tier cross_verified and records the batch under the key's section_census. A round-trip check (the key's own rows as the reviewer grade to 0 disagreements on every batch) was run before any reviewer. Census: 11 batches, one fresh-context reviewer each, 187 COBOL files, 100% coverage on all three corpora. 509 facts compared, 18 disagreements, every one a reviewer formatting slip, not a key or engine error: - 12: `verb` left null on HANDLE_ABEND rows. Kinds whose verb the kind fixes are now normalised in the grader (the CICS-name-padding precedent). - 4: a HANDLE ABEND label shifted into target_kind (ruled key_correct: COACTVWC.cbl:264, COCRDSLC.cbl:250 are HANDLE ABEND LABEL(ABEND-ROUTINE)). - 2: the brief's `DD` placeholder copied literally for SYSUT2 (ruled key_correct; the brief now says <ddname>). No key fix was needed. The ledger now labels every populated section cross_verified. A reviewer note worth keeping: COSGN00C tests its READ's RESP against the numbers 0 and 13, not DFHRESP names, so it reads as "untested" to both the engine and the key. That is an agreed limitation, not a disagreement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g
Contributor
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3445.
The six answer-key sections added by the epic matched the engine exactly, but they were only the drafting model's reading:
This PR gets them read a second time, blind.
Tool:
tests/tools/cross_verify_sections.pyIt runs the
cross_verify.pycensus over per-file sections:cross_verify.py: grade, rulings and sign work the same way. Signing sets the section's*_validatedflag and tiercross_verified, and records the batch under the key'ssection_census.Census result
verbleft null onHANDLE_ABENDrows. The grader now fills in a verb the kind already fixes, following the CICS-name-padding precedent.HANDLE ABENDlabel shifted intotarget_kind. Ruledkey_correctfrom the source: COACTVWC.cbl:264 and COCRDSLC.cbl:250 are bothHANDLE ABEND LABEL(ABEND-ROUTINE).DDplaceholder copied literally where the ddnameSYSUT2belongs. Ruledkey_correct, and the brief now says<ddname>.cross_verified.Agreed limitation, from a reviewer's note: COSGN00C tests its RESP against the numbers 0 and 13 rather than
DFHRESP(...)names, so the engine and the key both call it "untested".The mainframe-ground-truth skill now documents the section census, including the reviewer slips seen and how each was handled. 475 mainframe and tool tests pass, and the ruff and mypy audits show no new findings.
🤖 Generated with Claude Code
https://claude.ai/code/session_017ZsVaAkb86P5r5JXDC2Y9g