Repository navigation
test(chaos) confluentinc#857: the fifth item is the group protocol, and the PC half is re-verified with a control arm - #486
Conversation
…nd the PC half is re-verified with a control arm The 857 register's fifth open item - a rebalance stall in MultiInstanceRebalanceTest.largeNumberOfInstances, PERIODIC_CONSUMER_ASYNCHRONOUS/UNORDERED - read as blocked on progress-tracker instrumentation that does not exist. It is not blocked, and it has not been since #444: withDiagnostic is wired on master (MultiInstanceRebalanceTest passes describeFleet), and the stall has been reproduced repeatedly since, not never. Both of those claims were still in the section, so anyone picking the item up inherited an investigation that had already finished. The mechanism is measured and lives in test-largenumberofinstances-residual-failures-measured-not-explained.md's EXPLAINED section: the chaos monkey restarts members faster than a join phase completes, a LeaveGroup sent while the member's own JoinGroup is pending is not answered until that phase completes, and one phase was observed held open for 17s. While it is open consumer.poll() returns nothing to any member, including survivors that never toggled - the FLAT count. No coordinator request was slow in any failing run, and every PC-side candidate was refuted by measurement. So the verdict is neither wedge nor harness artefact: the freeze is real and fleet-wide, it recovers when the phase completes, and what turns it red is the detector's 12s window closing inside a real protocol freeze. Master has moved through the revoke and close seam since that measurement (#451, #466, #468), so the PC half was re-asked rather than inherited. Two arms on ClosingMemberRebalanceIT differing by exactly one term - a 20s sleep at the top of BrokerPollSystem's doClose(), which holds a member not-polling and not-left, the defect shape itself. Predictions written before each run: clean master - predicted PASS - 5/5 pass, 35.4s, load average 5.1-7.1 sabotaged - predicted FAIL on the close bound - 5/5 fail, "expected to be less than: 10.0 but was 20.009590667", load average 13.0 The sabotage arm is what makes the clean arm mean anything: every property this test asserts already holds on master, so a green alone cannot say whether the test can see the failure - the VOID-replay trap #485 recorded. One finding that is new rather than a re-derivation: the sabotaged arm emits this section's own signature line verbatim - ZOMBIE_MEMBER/REBALANCE_BLOCKED, dwelling in PreparingRebalance for 15s. A PC-side hold produces it and so does a coordinator holding its join phase open with PC off the critical path, so that line does not discriminate between the two stories, and a future sighting has to be told apart by what the closing members' threads are in. Deliberately not run: a local replay of the capacity profile. It reproduces at 0 in 22 on an M2 desktop against about 1 in 15 on the Linux runner, so a green here would have carried no information, and the box was at load 5-14 throughout - a confound, not a control. No product code changes. The vetted marker carries a PROPOSED close for the fifth item only, per docs/inflight/AGENTS.md's owner gate for a stall-impact note; the fourth item and #119 are untouched. Composes with #473, which carries the durable write-up and the lane move and rewrites the paragraph this commit leaves alone. Co-authored-by: Claude Opus <noreply@anthropic.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ✅ jscpd (language-agnostic)
No new clones introduced by this PR. Powered by astubbs/duplicate-code-cross-check |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #486 +/- ##
============================================
+ Coverage 82.44% 82.93% +0.49%
- Complexity 1554 1563 +9
============================================
Files 96 96
Lines 5377 5380 +3
Branches 534 535 +1
============================================
+ Hits 4433 4462 +29
+ Misses 750 725 -25
+ Partials 194 193 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
[superseded - the quarantine lane is now empty] 🧪🔒 Quarantine Lane ReportThe superseded report, collapsed because it no longer applies
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its Updated for Superseded by a newer quarantine lane report. |
🟢 Throughput — OKThis branch measured about 10% faster than master, on the one test this measures. That is INSIDE this test's own run-to-run spread of about 17%, so read it as a reading and not as a result - re-running the same commit moves it by about as much.
Allowable range 🟢 ≥ 0.70 · 🟡 0.50–0.70 (about a 30% loss) · 🔴 < 0.50 (about a 50% loss) What the numbers mean, and what they cannot tell youThe one that gets misread. Why a shape and not a rate. A rate depends on which runner you drew. A shape does not: every test here processes a fixed number of records, so a runner twice as slow doubles the subject and the controls together and leaves their ratio alone. That is the whole trick, and it is why the reported rate is shown last and labelled as this machine only. Reading the comparison. By conservation, not by correction. Every test in this lane processes a fixed number of records, so within one run the ratio of one test's time to another's is invariant under machine speed — a runner twice as slow doubles both terms and leaves the ratio alone. There is no machine-index correction to be wrong, because nothing needed correcting. Per-method times, not class times. A class time is Reference is the median of 5 recent What this still cannot do. It removes machine-to-machine variance. It does not remove this test's own run-to-run variance, measured at about 30% on a single unchanged commit while its controls stayed within 5%. That is a property of the test, not of the comparison, and no arithmetic here can touch it — which is why the reference is a median and the bounds are deliberately coarse. 🟡 means look at this; only 🔴 is outside the measured spread. Since the previous push: ratio 1 -> 1.098, share 1.909 -> 1.673, rate 81532 -> 95256 (+16.8%). One push of difference sits inside this test's measured spread - read it as movement, not as a result. Updated for |
|
…dered-rebalance-stall
#473 promoted #473 merged while this branch was open. It deleted test-largenumberofinstances-residual-failures-measured-not-explained.md, having migrated the mechanism to docs/solutions/test-flakiness/, which is the correct outcome under docs/inflight/AGENTS.md - but it left this branch citing a file that no longer exists, and bin/check-file-refs.sh caught it. Repointed at the solutions write-up, which is now the durable owner, and reworded the two places that spoke of #473 in the future tense. The merge itself was clean; only the citation needed repair. Co-authored-by: Claude Opus <noreply@anthropic.com>
🧪🔒 Quarantine Lane ReportThe quarantine lane is empty - no Any earlier row on this PR asking for a Lane: non-gating; rules: see the Quarantine Audit check. No quarantined test changed outcome since the previous push. Updated for |
|
@claude review this |
|
Claude finished @astubbs's task in 3m 32s —— View job Review: docs-only correction to
|
…s test's calibration Review finding on #486, raised as "worth knowing, not a defect": ClosingMemberRebalanceIT's javadoc already records the identical sabotage shape and near-identical numbers under its Proven red block, which arrived on this branch with #473's merge some minutes after the entry here was written. Two records of one shape is how a duplicate starts drifting, and the repo's rule is to name the owner in the pointer rather than let a reader who found one stop looking. So the entry now says which record owns what: the javadoc owns what the test is calibrated to detect, and the rows here are an independent re-run on a later master answering a different question - whether the PC-side story still holds now. It also warns the reader not to read the agreement between them as one result cited twice, which is the misreading the two-places shape invites. No claim changed and no number moved. Co-authored-by: Claude Opus <noreply@anthropic.com>
#486 The scope note still said the async-unordered rebalance stall was blocked on progress-tracker instrumentation that did not exist. #486 shows it was never blocked: the diagnostic has been wired since #444, the mechanism was already measured and promoted to master by #473 (a join phase the coordinator holds open, during which poll returns nothing to any member, then recovers), and two claims in the family note were false against master. The PC half was re-verified with a one-term control arm after the revoke and close seam moved. Not a defect; struck from the list and from the replay queue. Claude-Session: 460f7df9-dcc2-4b00-a9f9-62f3a2c6d5e4 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…dered-rebalance-stall
#485 landed on master and re-vetted the same note the same day, so both sides rewrote the tail of `bug-857-family.md`'s single `inflight-vetted` marker. That marker is read with one regex and `head -1`, so a second marker would be silently ignored - the two findings had to become one. Resolved by hand, keeping both and neither side's wording lost: the shared prefix, then #485's PROPOSED close for the revoke-path deadlock LINE, then this branch's PROPOSED close for the FIFTH item. The one clause that genuinely conflicted was the running total - #485 says "the fourth and fifth items are still open", this branch's whole subject is that the fifth is not - so the total now reads "the fourth item", which is what both PROPOSED clauses below it then qualify. Audited both parents for silent losses per the merge rule rather than stopping at a clean tree: the only removals against master are that marker and this branch's own deliberate rename of the fifth item's heading. Nothing of #485's six corrected capture sections was touched. Co-authored-by: Claude Opus <noreply@anthropic.com>
|
@claude review this |
|
Claude finished @astubbs's task in 1m 14s —— View job Review: docs-only correction to
|
…s down to the run-length guard The poller-death fix squash-merged, and its branch had rewritten item 2 of the sweep gating list in process-candidate-ranking.md, which this branch had already moved into release-v6-scope.md. The conflict resolves to the pointer here and the rewritten item in the moved copy. #477 is ticked in tier 1, six of seven on master; #480 is the last, reworked off the wire and level with master. #486 settled the fifth 857 item as the group protocol and is green with the owner LGTM; one new fact from it is kept, that the zombie-member probe line does not discriminate a PC-side hold from a coordinator join phase. Claude-Session: 460f7df9-dcc2-4b00-a9f9-62f3a2c6d5e4 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… both PROPOSED closes and the withdrawal in one marker #486 landed its PROPOSED close of the fifth item on the same vetted marker line this branch edits to withdraw the fourth. Resolved by taking master's marker and re-applying this branch's one clause - the fourth item is withdrawn by #478 - so the marker now carries #485's deadlock-line close, #486's fifth-item close and this withdrawal. The fifth item's heading takes master's RESOLVED form; the CORRECTED section this branch adds is unchanged. Claude-Session: 460f7df9-dcc2-4b00-a9f9-62f3a2c6d5e4 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Two more lines of the burn-down land on master on 2026-09-08: the shard-displacement orphan is known unreachable (#483, whose review round widened the regression test to every ordering mode), and the fifth 857 item is the group protocol (#486, carrying the PROPOSED close for the owner). The release-note draft said #408 adds the decline. It no longer does: #466 refuted declining as the fix by experiment, so the decline is only the deadline fallback, and what #408 now carries is the measurement of the bound and the held decline seam - the reading its reconciliation with master recorded today. Claude-Session: 460f7df9-dcc2-4b00-a9f9-62f3a2c6d5e4 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ster The fourth 857 item's withdrawal landed as a squash on 2026-09-08, after taking #486's marker change from master so the register's vetted line carries both PROPOSED closes and the withdrawal together. The burn-down records it merged in both places it is named. Claude-Session: 460f7df9-dcc2-4b00-a9f9-62f3a2c6d5e4 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Serves #119 (confluentinc#857).
Description
The 857 register's fifth open item - a rebalance stall in
MultiInstanceRebalanceTest.largeNumberOfInstances,PERIODIC_CONSUMER_ASYNCHRONOUS/UNORDERED-still reads as blocked on progress-tracker instrumentation that does not exist. It is not blocked,
and it has not been since #444. Two claims in that section are false
against master, and they are why the item kept being picked up as live work:
ProgressTracker.withDiagnostic(...)was not wired there" - it is wired.MultiInstanceRebalanceTestpassesdescribeFleet(allPCRunners)to it.about one run in fifteen, most recently 4 in 60 with the coordinator loggers raised.
The mechanism is already measured, in
docs/solutions/test-flakiness/large-instances-residual-is-a-join-phase-held-open-by-churn-2026-09-05.mdThis PR does not re-derive any of that. It corrects the register so the next reader is not sent
to chase a finished investigation, and it re-verifies the one part that could genuinely have
changed.
The verdict, in the terms the item was posed in
Neither a wedge nor a harness artefact. The freeze is real and fleet-wide - while the coordinator
holds a join phase open,
consumer.poll()returns nothing to any member, including survivorsthat never toggled, which is the
FLATcount - and it recovers when the phase completes, so nowork is stranded and no instance stays wedged in the group. No coordinator request was slow in any
failing run. What turns it into a red is the detector's 12s no-progress window closing inside a real
protocol freeze.
The PC half, re-verified with a control arm
Master has moved through the revoke and close seam since that measurement
(#451, #466, #468), so
"PC holds nothing" was worth re-asking rather than inheriting. Two arms on
ClosingMemberRebalanceITdiffering by exactly one term - a 20s sleep at the top ofBrokerPollSystem'sdoClose(), which holds a member not-polling and not-left, the defect shapeitself. Predictions written before each run.
doClose()expected to be less than: 10.0 but was 20.009590667(load average 13.0)The sabotage arm is what makes the clean arm mean anything: every property this test asserts already
holds on master, so a green alone cannot say whether the test can see the failure - the VOID-replay
trap #485 recorded.
One finding that is new rather than a re-derivation: the sabotaged arm emits the fifth item's own
signature line verbatim -
ZOMBIE_MEMBER/REBALANCE_BLOCKED: dwelling in PreparingRebalance for 15s.A PC-side hold produces it, and so does a coordinator holding its join phase open with PC off the
critical path. That line therefore does not discriminate between the two stories, and a future
sighting has to be told apart by what the closing members' threads are actually in.
What was deliberately not run
A local replay of the capacity profile. It reproduces at 0 in 22 on an M2 desktop against about 1 in
15 on the Linux runner, so a green here could only have carried no information, and the box was at
load 5-14 throughout - a confound, not a control. Reproducing it needs the Linux runner and
bin/exp-measure-large-instances-failure-rate.sh, which #444 already did.Scope and composition
No product code changes; the sabotage was reverted before the commit. The vetted marker carries a
PROPOSEDclose for the fifth item only, perdocs/inflight/AGENTS.md's owner gate for astall-impact note - the fourth item and #119 are untouched, and neither isthe quarantine registry, whose entry already states the measured mechanism correctly.
#473 merged mid-flight and is now merged into this branch. It rewrote
the closing paragraph of this section and deleted the note that used to hold the measurement; this
PR changes the heading, the vetted marker, and appends after that paragraph, so the two do not
restate each other. The merge was clean; the only repair it needed was a citation
bin/check-file-refs.shcaught, repointed at the solutions write-up that replaced the deleted note.Checklist
docs/inflight/bug-857-family.mdis the changeN/A - no new test; the existingClosingMemberRebalanceITwas run as the two-arm experiment, unmodified, and no assertion was weakeneddocs/inflight/working note - N/A - the note this PR updates is the tracking record, and apr-note would restate itce-simplifyhas no code to work on; asking@claude review thison the PR instead of a localce-code-review