Repository navigation
Referee panel, round 2: re-score on current main, referee-majority column, Che4z over LSP, nightly run #4438
Description
Follow-up to #4377, whose first measured pass is done (tooling #4384; scorecard in #4377's comment; every engine and key finding fixed: #4302, #4305, #4306, #4391, #4392, #4393).
Remaining
- Re-run the panel on current main. The published numbers predate the five engine fixes and the key corrections, so re-score and post the new scorecard.
- Add a referee-majority column: engine vs 2-of-3 referees, per fact channel.
- Drive Che4z over LSP (
documentSymbol+ diagnostics) instead of the 2.5.1analysisCLI, which exits 1 on any program with a CALL and overflows on large ASTs. That cost it about a quarter of the members. - Run nightly, not as a PR gate. JVM start-up plus the 5-minute grammar generation make it a poor PR gate.
- Optionally, upstream reports for tree-sitter-cobol-enterprise (
CBL/PROCESScards,ENTRY ... USING, DBCS names) and Cobrix (pic xXXXXXread as 5 bytes).
Policy (unchanged): referees stay test and validation tooling only. They are never vendored, imported by the product or shipped.
🤖 Generated with Claude Code
Activity
squid-protocol commentedon Oct 5, 2026
Note 2026-10-05: running the re-score internally is fine, but publishing new comparative results (scorecards or docs pages naming other tools) is on hold pending a review. Post numbers here only once that is cleared.
referees: engine vs referee-majority column in the scorecard (Part of #…
squid-protocol commentedon Oct 18, 2026
Item 2 (referee-majority column) is merged in #5409. Items 1 (re-score on main; publishing on hold per owner note 2026-10-05), 3 (Che4z over LSP) and 4 (nightly run) remain, so this stays open.
squid-protocol commentedon Oct 23, 2026
Churn run
4438-20261023T0152Z(host nyx) ended without a PR. No PR: referees not installed on box, publishing on hold; item 2 already merged in #5409. Handoff at handoffs/churn-4438-handoff.mdche4z referee: pass UTF-8 flags to the LSP java subprocess (Part of #…
Che4z referee: drive the language server over LSP documentSymbol (Par…
Referee panel: one-command nightly runner, scratch results only (Part…
ci: nightly referee-panel workflow (schedule + dispatch only, content…
referee panel workflow: copy the Cobrix classpath jars into the cache…
Drop the referee-panel.yml change (needs the owner: workflow edits ar…
test_signal_probes_4108_slice4: pin the zip timestamp so parametrize …
referee panel: first dispatch fixes (cached Cobrix classpath, referee…
referee-panel: copy Cobrix's Maven jars into the referee cache and re…
squid-protocol commentedon Oct 24, 2026
Churn run
4438-20261024T2034Z(host melek) ended without a PR. No PR: referee-majority column already in main (#5409); rescore saved in handoff-4438-rescore-20261024.md; publishing on owner hold (2026-10-05); issue stays opensquid-protocol commentedon Oct 25, 2026
Churn run
4438-20261025T2059Z(host nyx) ended without a PR. No PR: remaining items wait on owner (publishing hold, upstream reports)