tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially

[kyle-sexton]

Two contract suites fail under scripts/run-plugin-tests.sh --jobs 4 in CI, pass serially and pass locally, and no hypothesis so far explains why. Both are on the serial allowlist as a fail-toward-running measure so the parallel runner can land; this issue holds the debt so the allowlist entries do not become permanent by default.

Evidence

Run 33787030286 on melodic-software/claude-code-plugins, plugin-gate job, Run plugin contract tests step, branch ci-perf/phase-2-scripts, head 68f18b483, runner ubuntu-24.04.

plugins/discovery/agents/tool-honesty.test.sh — two assertions failed:

FAIL - explorer.md: the return payload has no `persistence:` field — a completed run whose write was refused cannot be told apart from one that never ran
FAIL - researcher.md: the return payload has no `persistence:` field — a completed run whose write was refused cannot be told apart from one that never ran

The identical assertion passed for intent-tracer.md in the same run. The check is body "$agent" | grep -q '^persistence: ' at plugins/discovery/agents/tool-honesty.test.sh:147. Check 6 immediately below it (scope_as_received/topic_as_received) uses the same body() helper on the same files and passed for all three, so body() produced output at least once per file.

plugins/work-items/tools/work-item-tracker/conformance/bindings/jira.test.sh — case [4] failed:

FAIL: [4] conformance --binding jira exit 0 — expected 0 got 1

Case [5] is the same conformance run under a PATH shim that makes gh and curl exit 1, and it passed. Case [4] is therefore the only one of the pair that runs with real network tools on PATH.

What is not the cause

• Not a pre-existing failure. main's own most recent run at the same parent SHA (33706296925) passed both suites. That run failed only plugins/performance/scripts/ab.test.sh, a wall-clock benchmark already on the serial allowlist.
• Not a pipefail plus grep -q SIGPIPE race. This was the first hypothesis: tool-honesty.test.sh sets set -uo pipefail, and grep -q exits on its first match, which can leave awk writing into a closed pipe and turn a successful match into a non-zero pipeline status. Two hundred iterations of the exact pipeline under load produced zero failures, and the file body fits inside the pipe buffer, so awk finishes writing before grep -q can exit. Rejected.
• Not visible resource exhaustion. The run's log was searched for fork, Resource temporarily unavailable, Cannot allocate, out of memory, No space left, too many open files and Killed. Only ordinary test prose matched. If a fork or allocation failure is behind this, it left no message.
• Not reproducible locally on a developer box. Both suites pass on Windows Git Bash on this branch. That box reports 32 cores against the hosted runner's 4, so it cannot reproduce contention on a 4-vCPU runner; local reproduction is the wrong instrument here and a local pass is not evidence.

Standing hypothesis

The suites are spawn-bound, and four concurrent ones on a four-vCPU hosted runner is the aggressive end of the range docs/topics/ci-perf/research/PROFILE-ccp-scripts.md predicted. The jira case is the more legible half: it is the one case in that file that runs with real gh and curl reachable, so network or rate-limit contention under concurrency is a plausible mechanism. tool-honesty.test.sh has no clock, no network and no shared state, and remains unexplained.

Done means

Either the mechanism is identified and the underlying defect fixed, at which point both entries come off scripts/run-plugin-tests-serial.txt, or this issue records why the entries stay with the reason stated as something other than "unexplained".

A --jobs 3 measurement is planned on the same branch and will be recorded here: if the failures disappear at three concurrent suites, that is evidence for the contention hypothesis and the job count, not the allowlist, is the lever.

Refs melodic-software/github-iac#385

[kyle-sexton]

A third suite joins the pattern, and the job count now looks like the lever rather than the allowlist.

Third occurrence. Run 33794273916, plugin-gate, --jobs 4, after the first two suites had been serialised:

FAIL: plugins/work-items/tools/work-item-tracker/adapters/local-markdown/list-sub-items.test.sh
FAIL: [9] list-sub-items --state closed keeps only C2 — expected local-markdown:local/markdown#3 got ''

Case [8] immediately above it, the same call with --state open, passed in the same run. The case has no clock in it: it reads a local-markdown fixture through jq and compares ids.

The shape all three share. In every failure an external command produced EMPTY output on a deterministic path: an awk read of an agent markdown file (tool-honesty.test.sh), a jq read of a fixture (list-sub-items.test.sh), and a conformance run with real network tools on PATH (jira.test.sh). None of the three is timing-based, all three pass serially in the same job on main, and all three pass locally. That reads as one symptom with three faces rather than three flaky suites, which also means serialising each one as it appears would grow this list by a suite per run while hiding the cause.

The --jobs 3 result. Run 33797359110 is green on every lane with the contract corpus at three concurrent suites. Step wall times from the Run plugin contract tests step, measured the same way on both sides:

Setting	Run	Step wall time	Result
serial	33656901629	572 s	green
--jobs 4	33787030286	228 s	red
--jobs 4	33794273916	240 s	red
--jobs 3	33797359110	312 s	green

So three concurrent suites keeps most of the gain (1.8x against serial) and is the only parallel setting that has produced a green run. ci.yml now passes --jobs 3.

This is evidence for the contention hypothesis but not a diagnosis: nothing yet explains WHY a fourth concurrent spawn-bound suite makes an unrelated awk or jq return empty instead of failing loudly. Until that is known, the two entries on the serial allowlist stay, and this issue stays open. If the mechanism turns out to be a property of the runner rather than of the suites, the right fix may be neither the allowlist nor the job count.