Repository navigation
test(probe): ambient flight recorder - every broker IT failure arrives pre-diagnosed - #86
Merged
Merged
Conversation
…same-repo only) Optional, non-gating fast feedback that runs unit, integration and performance as parallel matrix jobs on a self-hosted many-core runner (label `highcpu`), guarded to same-repo PRs so fork code never executes on it. The GitHub-hosted gate in maven.yml stays the required check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…rkflow # Conflicts: # .github/actionlint.yaml
…-only The high-CPU self-hosted matrix now runs unit + integration + performance + mutation as parallel jobs (a strict superset of the mac runner, which is offline indefinitely). The mac workflow is switched to workflow_dispatch-only so it stops queuing dead jobs on every PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
failingHttpCall drove an HTTP call at a dotless bogus hostname ("xxxxxxxxx") and asserted the failure
was a DNS *resolution* error ("failed resolve"). That only holds where the name fails to resolve - on any
network with a local resolver + search domain (e.g. a Pi-hole LAN) the name resolves and the failure mode
becomes "connection refused", so the test fails there while passing on GitHub's public DNS.
Point the bad request at a closed local port (127.0.0.1:1) instead: an immediate, deterministic
"connection refused" on every environment, no DNS lookup at all (and faster than one). Keep the real
failed==true check; assert the deterministic "connection refused" message. Found bringing up the
self-hosted high-CPU runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…with failingHttpCall) The failingHttpCall fix (127.0.0.1:1 -> connection refused) also flipped this sibling test through the shared getBadRequest() helper, so its assertion is updated to match. Caught by ce-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Load was ~18/32 with 34GB free - the box is I/O-bound (cold cache), not CPU-bound - so use all the cores. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…ateCommittedOffsetIT Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…epo guard (ce-review P2) The if: guard lives in the PR-head file, so the real RCE backstop is the repo Actions setting "require approval for all outside collaborators". Documented in the workflow header + referenced SELF_HOSTED_RUNNER.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Unit forkCount 1C->2C, Integration 8->16 (probe the oversubscription ceiling). Mutation PIT threads 2->16 (CPU-bound, stays in the matrix). Performance split into its own job that needs the matrix, so the throughput benchmark measures on a quiet box (solo 1m54 vs 7m18 when it shares the box). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…formance Doubling was flat (unit 2C=1C) to worse (integration 16 slower than 8), so back to the knee. Performance returns to the concurrent matrix - it is a correctness gate here; noisy timing is fine and isolation wrecked total wall-clock (it queued behind a 17-min mutation). Mutation keeps threads=16. Accurate benchmarking -> separate on-demand run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Unit/Integration now also enable JUnit method-parallelism (-Dparallel-tests=true) on top of forking, forks reduced so forks x threads ~= cores - tests whether thread-parallelism is still flaky after recent fixes (revert to fork-only if red). ci-mutation-test.sh: on a PR (GITHUB_BASE_REF) mutate ONLY changed core classes, skip if none - the full internal.* sweep has never completed. Full sweep still runs for push/nightly (no base ref). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…eads finding fork×threads (parallel-tests=true) was safe (threading no longer races - unit green; integration red was the known flaky PartitionStateCommittedOffsetIT) but no faster than fork-only, so back to 1C/8. PIT: grep no-match under set -euo pipefail made the changed-class skip exit non-zero (0m19 red); wrap grep with || true so it exits 0. Recorded the thread-parallelism-vs-forking finding in inflight next to the shelved Step-2 question. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…bserve a green board TEMPORARY - not a fix. Disables the known-flaky PartitionStateCommittedOffsetIT.committedOffsetRemoved (awaitility timeout, in inflight, fails on GitHub-hosted too) purely to see a fully-green highcpu run. Revert and harden the test properly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Now that ci-mutation-test.sh scopes PIT to changed classes on PRs (GITHUB_BASE_REF), it is fast enough for the 2-core hosted runner - so mutation runs on GitHub-hosted as well, not just the self-hosted box. Non-gating (continue-on-error). fetch-depth: 0 so the changed-class diff resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…already passed Each GitHub-hosted test job, before running its slow 2-core suite, polls whether the self-hosted highcpu equivalent for the same commit already PASSED (ci-concede-check.sh). If so it skips and goes green; the hosted job stays the required gate but finishes faster when grumpy wins. Never depends on the self-hosted runner: offline/queued/slow/failed -> run the tests normally. Bounded wait, continue-on-error so a check bug can never fail the gate. Caveat documented: conceding trusts the self-hosted env (env-specific failures could be missed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
… the green-board experiment) Reverts the temporary experiment disable. The test is still the known flaky awaitility timeout (tracked in inflight) - harden it (adaptive await / underlying timing) rather than leave it disabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
The check name is "<workflow> / <job>", and both halves carried "High-CPU"
plus verbose filler ("Build & Test - ...", "self-hosted, optional"), pushing
past the PR merge-box truncation width.
- workflow name: "PR High-CPU Fast Feedback" -> "highcpu"
- job name: "Build & Test - <suite> (high-CPU self-hosted, optional)"
-> "<suite> (optional)" (keeps the non-gating signal)
Result: "highcpu / Unit (optional)", "highcpu / Integration (optional)", etc.
- longest ~35 chars, well under the cutoff.
Kept the concede matcher in sync (it resolves the highcpu run/jobs by name):
bin/ci-concede-check.sh WORKFLOW default -> "highcpu", JOB_PREFIX -> "${SUITE}"
(the job is now named "<suite> (optional)", so a suite-name prefix still
uniquely matches). Added a keep-in-sync comment on the job. Updated the
SELF_HOSTED_RUNNER.md reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
… behind Mutation's slow cancel A single workflow-level concurrency group meant every new push's jobs waited for the ENTIRE previous run to complete before starting. cancel-in-progress signalled the old run immediately, but its Mutation (PIT) maven/JVM tree rides out the runner's full cancellation grace (~5 min) before force-kill - observed on PR #80: push at 22:41:05, old Mutation force-killed 22:46:06, all four new jobs started 22:46:09. Head-of-line blocking on the one job nobody needs fast feedback from. Job-level per-suite groups (highcpu-<suite>-<branch>) mean a new push supersedes each suite independently: Unit/Integration/Performance predecessors are finished or die fast, so they start within seconds; only Mutation queues behind old Mutation, which is fine for a 30-minute advisory job. No change to runner usage. actionlint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…ner' in comments Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…robes (W1 skateboard) Implements Phase 1 of the Chaos Pain Suite design (branch docs/chaos-pain-suite-design; plan under docs/plans/). Additive-only on the uber experiment composition so it transplants onto the rebased-#29 slices as a file copy - zero edits to #29-owned files. - ChaosConductor: seeded, replayable action scheduler (STOP_DRAIN / STOP_NO_DRAIN / RESTART / JOIN_NEW) with EnumMap-deterministic weighted picks, a timestamped action timeline logged every run (the autopsy's first artifact), a join-after-stopDrain bias knob (the zombie-drain collision the calibration targets), and its own per-instance state machine over ManagedPCInstance - including raw closeDrainFirst() for DRAIN stops, which the managed API doesn't expose. - ProgressProbe: independent SLO samplers - progress watermark (30s, with tail slack for legitimately-sleeping heavy records), rebalance-dwell zombie probe (60s: >> seconds-healthy, << 5-min zombie; keyed on protocol-unresponsiveness, NOT partition-holding, which is correct drain behaviour), drain-completion bound (150s, must exceed the heavy tail), plus the end-of-run correctness ledger (no loss ever; duplicates bounded per disturbance, capacity-shaped). - ChaosChurnStormIT (W1): 12->16 fleet, 80 partitions, 100k messages with heavy-tailed work (1-in-4000 records sleeps 90s) - the heavy tail is what stretches drains into the probe's discrimination window, since the zombie-drain defect freezes the group for the DURATION of a drain. - ChaosConductorPlanIT: seed-determinism regression (same seed = same plan). - @tag(chaos) excluded from all default suites (pom default now performance,chaos; comment documents the multi-group override gotcha); chaos-pain.yml workflow_dispatch runs it on-demand on the highcpu runner (seed + reps inputs, artifacts uploaded, never gating). Validated: 3 green storm runs (500k and 2x100k msgs - zero loss, duplicates ~500-2700 vs allowances), determinism 2/2, default-exclusion verified (name-matched chaos test runs 0 tests without -Dincluded.groups=chaos). Calibration against the planted zombie-drain defect arm is the next commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…(portable bug-detector pack) Completes the suite for standalone landing on master: - ManagedPCInstance extracted from the confluentinc#857 investigation branch (PR #29, credited) - the multi-instance PC lifecycle harness; verified mainline-API-only. When #29 rebases, its harness slice simply drops its copy. - Suite files at calibrated state: non-interruptible heavy tail (PC's close force-interrupts stragglers ~11s, so interruptible sleeps cannot stretch drains into a discriminable window), rebalance-dwell ZOMBIE bound 15s set from same-seed A/B measurement (healthy peak 6.7s vs defect 20.1s), peak signature logging, hotter conductor (0.5-1.5s ticks, join-after-drain bias 0.9). Calibration record (see the design doc on docs/chaos-pain-suite-design and plan on experiment/chaos-pain-suite-phase1): planted-defect arm 3/3 RED via ZOMBIE_MEMBER/REBALANCE_BLOCKED (dwell 15.6-15.9s), all-fixes arm 3/3 GREEN (4.5-8.9s), and PLAIN MASTER: RED via the same probe (15.6s) - this branch's own base carries the drain-zombie bug (fixed on PR #80), and the suite finds it blind. That is the point: a portable detector, excluded from every gating suite (@tag chaos), temporarily mergeable into any PR to expose fix-absent bugs and verify fixes. Run on demand: -Dincluded.groups=chaos [-Dchaos.seed=N], or the chaos-pain.yml workflow_dispatch on the highcpu runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…it; nothing is 'planted'/injected 'Planted' (mutation-testing jargon) wrongly implied synthetic fault insertion. The defect arm is git archaeology: the genuine historical bug in its original code, selected by basing on the pre-fix commit. The only engineered element is the sensitizing workload (non-interruptible heavy tail) that makes the real defect's freeze long enough to discriminate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
… stagnation Detects protocol-INVISIBLE stalls: group STABLE, heartbeats + polls flowing, no rebalance pending (so the broker's 5-min eviction clock never starts), yet a partition's lag is real and its committed offset stops moving - the 'locks forever until manual restart' class from the confluentinc#857 reports, which the Class 1 zombie probe cannot see. Mechanism: every 5th sampler tick, admin listConsumerGroupOffsets vs listOffsets(latest); violation CLASS2_STALL/LAG_STAGNATION when lag >= 50 and the committed offset has been stagnant > 150s. Peak (maxLagStagnation) logged every run. Calibration so far (test-the-test discipline): - First healthy run FIRED at 151s: a real measurement, not a probe bug - a hard stop interrupts a heavy record mid-dwell and at-least-once re-runs it fresh, legitimately blocking that partition ~2 chained dwells. Fix per the calibrate-the-workload lesson: heavy dwell 90s -> 45s (the zombie window only needs drains occupied; close bails at ~11s regardless), so a 2-dwell redelivery chain (~90s) fits under the 150s bound. Arithmetic recorded in both javadocs. - Re-validated: healthy arm 2/2 GREEN (peak lag stagnation ~39s, ~4x margin); defect arm still RED on ZOMBIE_MEMBER with unchanged dwell signature (15.7s vs 15.6-15.9s at the old 90s dwell). - RED calibration of this probe awaits the W4 revoke-under-work trigger scenario (next Phase 2 step). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
… the pom The gating scripts hardcode -Dexcluded.groups (overriding the pom default), so adding 'chaos' only to the pom let ChaosChurnStormIT leak into the required Integration Tests check - where it promptly went RED on this branch's master base (the detector detecting; but it must not gate). Same class of bug the quarantine-lane PR #84's contract test now pins for the 'quarantined' group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…s pre-diagnosed AmbientProbeExtension (JUnit 5, registered on BrokerIntegrationTest so all ~104 broker ITs inherit it with zero per-test changes) runs ProgressProbe in a new observer mode: group rebalance-dwell + per-partition committed-offset lag stagnation, sampled from admin-client reads only. On test failure it logs an AUTOPSY block - violations, peaks, and a frozen-partitions table (committed/end/lag/stagnation) - so a generic Awaitility timeout becomes 'zombie member blocked rebalance 15s' or 'partition 21 committed frozen lag=1132' or 'probe clean - fault is in the test'. Observer mode NEVER fails a test; the chaos suite remains the only place probe violations gate. ProgressProbe refactor is additive + DRY (extension contains zero sampler logic): observer-mode construction path; group id via Supplier (tracks GroupOption.NEW_GROUP switches); watermark maps re-keyed Integer -> TopicPartition (removes a latent cross-topic partition-number collision); multi-topic sampling when no topic filter is given. Chaos construction path unchanged in signature, thread name, log prefixes, bounds, and cadence. Escape hatches: -Dambient.probe=off (global), @NoAmbientProbe (class/method). Verified: W1 chaos run on master base bit-identical (RED via ZOMBIE_MEMBER, dwell peak 15.665s, own peaks line unchanged) AND produced the extension's first real autopsy - naming the zombie plus a frozen-partitions table; silence-on-success confirmed on a green IT (zero ambient output); compile clean on Java 8 API surface. Implemented by a subagent in an isolated worktree; reviewed, verified and committed by the orchestrating session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ✅ jscpd (language-agnostic)
No new clones introduced by this PR. Powered by astubbs/duplicate-code-cross-check |
✅ SpotBugs ReportNo bugs found (new bugs only — baseline from base branch excluded). |
3 of 5 tasks
Addresses the Claude auto-review findings on PR #83: 1. chaos-pain.yml: workflow_dispatch inputs (seed, reps) now pass through env vars instead of ${{ }} template substitution inside run: - raw interpolation executes attacker-shaped input as shell on the self-hosted runner (the review flagged seed; reps had the same vector). Also ${SEED_ARG:+...} quoting per shellcheck. 2. ManagedPCInstance.run(): try/finally MDC.remove - work-stealing pool threads are reused across instances, so the Runner-N tag leaked into later instances' setup logs between restarts. Finding 3 (loose ledger duplicate allowance) intentionally unchanged: it is the documented vacuous-bound guard; the evidence-based tightening is rostered in docs/inflight.md behind revoke-event instrumentation. Verified: test-compile clean, actionlint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
…hecklist gate (#97) Resolutions: newer #97 checklist wording in CHANGELOG/README/AGENTS (keeping this branch's in-thread review-response policy bullet alongside); ci scripts keep the chaos-inclusive exclusion list per the quarantine contract test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
The PR-checklist gate (now on master via #97) requires it - and the suite is changelog-worthy by house style (CI/dev-facing entries are in). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
…unds 3-5 follow-up The chaos-pain job deliberately excludes @Quarantined scenarios (decision in c3ba4e1: the known #80-pending W1 RED must not drown the tripwire signal), which currently means it selects ZERO tests. The review's two outstanding asks, now addressed: - chaos-pain.yml timing summary: prints a loud 'ZERO chaos tests selected - this run measured NOTHING' banner (instead of an empty table impersonating a real GREEN) when no failsafe reports exist; header comment documents the quarantine interaction and the interim window. - AGENTS.md chaos section + docs/inflight.md: the CI job is no longer described as equivalent to the local full recipe - the quarantine exclusion and the temporary zero-test state are called out explicitly. Coverage returns when the W4 variants (#85/#87) or the #80 fix land. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
…ts/chaos-ambient-probe # Conflicts: # CHANGELOG.adoc
|
Claude finished @astubbs's task in 3m 37s —— View job Claude finished @unknown's taskTasks
ReviewThis push is the review-hardening round (ffbe9bc) responding to the earlier 8-reviewer pass. I verified each claimed fix against the actual diff (
Residual, non-blocking observations (didn't rise to the level of the prior P1-P3s, flagging for awareness rather than as required fixes):
None of these are blockers — the core contracts (never fails a test, silent on green, no cross-topic collision, no race on the primary teardown path) all hold up under reading. I was not able to run |
'Temporarily merge the chaos branch into the PR under test' only made sense while the suite lived on an unmerged branch. Once this PR lands the suite is on master, so the recipe is: on the fix PR's branch (merge master in if it predates the suite), run pre-fix vs fix commits and watch RED->GREEN. Fixed in AGENTS.md + the W1 class javadoc; the PR body is updated in the same pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
5 tasks done
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
…s dispatch-only Review of today's runner meltdown: independent self-hosted workflows can't see each other's load, and the per-PR trigger semantics (cancel superseded) fought chaos-pain's dispatch semantics (queue, never kill). Split cleanly: - bin/chaos-test.sh: shared runner - the chaos mvn invocation, timing summary and the zero-test quarantine banner (emitted on EVERY exit via trap, so a RED run still gets its autopsy summary). Inputs via env only. - pr-highcpu-fast-feedback.yml: new 'Chaos Pain Suite' matrix entry - the per-PR ambient tripwire, sharing the lane's same-repo guard and per-suite superseding. NOT optional (continue-on-error false, bare check name): a chaos RED is a real finding and shows red. - chaos-pain.yml: workflow_dispatch only (seed/reps hunts), queue-never- kill preserved; delegates to bin/chaos-test.sh. - AGENTS.md / docs/inflight.md / CHANGELOG+README bullet updated. Verified: bash -n, actionlint both workflows, copyright scan clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
5 tasks done
… reported green Review round 6 caught a real regression from effa942: an EXIT trap's own last command becomes the script's exit status, and everything in emit_summaries is made to succeed - so bin/chaos-test.sh exited 0 even when mvnw verify failed, turning a real chaos RED green at both call sites (the non-optional highcpu lane entry included). Exactly the silent-success pattern five prior rounds stamped out. Fix: capture $? first, re-exit with it after emitting summaries. Verified with stub harness: failing run propagates exit 1 (banner still emitted), passing run exits 0. Also: CHANGELOG/README no longer describe the highcpu lane as blanket continue-on-error - the Chaos Pain Suite entry is deliberately not (round-6 minor note). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
astubbs
added a commit
that referenced
this pull request
Jul 31, 2026
…work - head start on #83 landing Non-trivial resolutions (base evolved past what this branch extracted): - ChaosScenarioBase helpers upgraded to base's new W1 epilogue semantics: settleFleet waits out stopAsync close-pending, settleRun takes/kills the pcExecutor, assertScenarioSlos uses the full replay command in messages, adds the unclassified-failure-cause canary sweep, and uses conductor.getDisturbanceCount() - W1 keeps its slim helper-calling body + base's @Quarantined + replayCmd - W4 adopts the same replayCmd pattern for message parity - W4 plan tests ported from removed planActions to planTicks/TickDraws (base's single-draw-path rework); ChaosConductorPlanIT 4/4 green - ManagedPCInstance: extraConsumerProps merge re-seated in base's restructured run(); Config field unchanged - AGENTS.md: dropped this branch's chaos signpost bullet (base now has a full Chaos Pain Suite section); added W4 + ChaosScenarioBase to it - inflight.md: chaos Phase 2+ roster kept, updated to post-merge reality (cooperative lever -> PR #87, ambient probe -> PR #86); dropped the fork-header roster entry (base's copyright work already flipped them) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018igSVt74wPQAbRNHkXS4R8
…dencies' Two fixes for the newly-required check: - synchronize added to the trigger types: required checks are evaluated per head SHA, and without it a plain push produced no run - the check sat 'Expected - waiting for status' forever, blocking merges. - Job renamed 'Check Dependencies' -> 'Check PR Dependencies' (clearer: it gates stacked-PR 'depends on #N' ordering, not library dependencies). Ruleset context updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
…ts/chaos-ambient-probe # Conflicts: # CHANGELOG.adoc
|
Claude finished @astubbs's task in 4m 9s —— View job Claude finished reviewing this PRTasks
ReviewI diffed Verified sound:
Residual, non-blocking observations (same class of issue as the prior pass — not asking for changes, flagging for awareness):
None of these are blockers. The core contracts this round of hardening targeted — never fails a test, silent on green, no cross-topic collision, sampler doesn't race the primary teardown path — all hold up under direct reading of the current diff. I did not run |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
…e javadoc precision Automatic-review observations on the ambient probe, dispositioned: - KafkaClientUtils.close(): null the admin field in a finally so a throwing admin.close() cannot leave a stale reference behind adminIfOpen() - AmbientProbeExtension javadoc: record that the @BeforeEach-throws claim is verified against the pinned junit-jupiter-engine bytecode (a @beforeeach failure branches past the AfterTestExecutionCallback phase), and document the one contained edge where the safety-net stop runs after the admin close Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4ENhXsAkPXvuK8gGsfxCo
Owner
Author
Response to the automatic review observations - 76ec2ecAll four residual observations from the latest automatic review dispositioned: 1. Javadoc claim " A 2. Sampler-vs-close race in the 3. 4. Fixed per-mode sampler thread names - acknowledged, declining. Broker ITs run under failsafe with per-class forking; two ambient samplers never coexist in one JVM in this repo's configuration, so the name collision is unreachable in practice and not worth the churn. 🤖 Generated with Claude Code |
…-probe # Conflicts: # AGENTS.md # CHANGELOG.adoc # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/chaostests/ChaosChurnStormIT.java # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/chaostests/ChaosConductor.java # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/chaostests/ChaosConductorPlanIT.java # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/chaostests/ProgressProbe.java # parallel-consumer-core/src/test-integration/java/io/confluent/parallelconsumer/integrationTests/utils/ManagedPCInstance.java
|
Claude finished @astubbs's task in 1m 57s —— View job Tasks
ReviewThis synchronize push is two merge commits ( The four dispositions from the last review-response comment all check out in the code:
The merge conflict resolution in Verified correct on this pass:
No new issues found in this push. I don't have anything to add beyond what the last two review rounds already surfaced and dispositioned. As in the prior passes, I could not run |
astubbs
added a commit
that referenced
this pull request
Jul 31, 2026
The 7-line seed + replay-command construction was cloned between W1's ChaosChurnStormIT and the W4 driver (flagged by the PR duplication report). One copy in the base keeps the replay contract - a raw CI log must be self-sufficient to reproduce - from drifting between scenario families. Ledger: drop the ambient-probe bullet (PR #86 merged); record the four parallel-suite unit flakes seen this session under CI reliability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
astubbs
added a commit
that referenced
this pull request
Jul 31, 2026
…cenario - PC's first cooperative-assignor coverage (#87) Extract the two-phase storm/quiet W4 driver into AbstractRevokeUnderWorkScenario: the eager IT becomes a thin subclass, and ChaosRevokeUnderWorkCooperativeIT runs the identical scenario with CooperativeStickyAssignor as the single experimental variable - the codebase's first end-to-end exercise of PC under incremental rebalancing. Both arms GREEN across all seeds: backlog drained, correctness ledger balanced, zero probe violations. Findings (same-seed A/B, defect arm vs fixed arm): - Sticky drops revoke events ~6x vs eager (5-15 vs ~57 per run), refuting the going-in more-revokes hypothesis; per-event sharpness rises instead (partitions keep processing through rebalances). - The Class 1 dwell measurement does not discriminate arms under cooperative - eager-calibrated bounds must not be transplanted; a future W1-coop variant needs its own calibration. - No Class 2 trigger on either assignor: the probe stands as a calibrated tripwire (GREEN-side validated both assignors, RED-side awaits a real occurrence via the ambient probe, #86). Also dedups per the PR duplication reports: the seed/replay-command block is hoisted into ChaosScenarioBase (resolveSeed/replayCommand), shared with W1's ChaosChurnStormIT. Plan: docs/plans/2026-07-31-001-feat-chaos-w4-cooperative-variant-plan.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 tasks done
astubbs
added a commit
that referenced
this pull request
Aug 3, 2026
The plan was created and fully executed inside this PR, so its forward-looking half (requirements trace, scope boundaries, implementation units, open questions, risks) is now inert, while its diagnosis narrative duplicated the solutions doc almost line for line. Keeps the durable artifact and folds in the one thing only the plan carried: the classification verdict - what this was NOT (not the tight-timeout contention pattern, not a main-code bug, not already tracked) - which is the part most likely to save a future reader from the wrong fix. Repoints the two inbound references: the test javadoc and the inflight ledger now cite the solutions doc, which is the artifact that still makes sense in a year. README regeneration also picks up #86's ambient-probe changelog entry from the master merge. Test re-verified green after the javadoc repoint (2/2). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
astubbs
added a commit
that referenced
this pull request
Aug 7, 2026
Reviewing the new solution doc surfaced a hole in the checklist that same PR had just added: it said to check open PRs, which is a collision check. The PR that already solved something in your file is by definition merged, so searching only the open list finds nothing and feels like diligence - worse than not looking, because it produces false confidence. Same for issues: the useful ones are usually closed. Searching merged PRs BY TOUCHED FILE - not by keywords in their titles - finds, for this investigation: - #110, the sibling flake on this same producerTransactionLock, fixed four days earlier and the source of the control-arm method - #86, which introduced the ambient probe whose "probe clean" verdict this work had to qualify - #98, the backpressure test that only passed by racing its own setup - the sibling rule this doc deliberately does not merge with None would surface from a symptom search or from the open list. All three are now cited in the solution doc, with #80, #115 and #68 from the same family, and the confluentinc#803 / confluentinc#809 / confluentinc#833 cluster upstream. Adds to the doc what the prior-art search actually covered, including what it did NOT (the highcpu and quarantine lanes), so the next reader knows where the gaps are instead of assuming there are none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Was stacked on #83 (Chaos Pain Suite, now merged); base is master.
Ambient probe: a flight recorder for every broker integration test.
AmbientProbeExtension(JUnit 5, registered onBrokerIntegrationTest- all ~104 broker ITs inherit it, zero per-test changes) runs the chaos suite'sProgressProbein a new observer mode under every test: group rebalance-dwell + per-partition committed-offset lag stagnation, sampled purely from admin-client reads.On test failure it logs an autopsy block - violations, peaks, and a frozen-partitions table (committed / end / lag / stagnation per partition) - so a generic Awaitility timeout arrives pre-diagnosed: "zombie member blocked rebalance 15s" vs "partition 21 committed frozen lag=1132" vs "probe clean - fault is likely in the test itself". Exactly the material that historically took a manual debugging session per flake (see the PartitionStateCommittedOffsetIT saga in
docs/solutions/test-flakiness/).Observer mode never fails a test. The chaos suite remains the only place probe violations gate (its thresholds are chaos-calibrated). On green tests the probe is silent (DEBUG one-liner). Escape hatches:
-Dambient.probe=off(global),@NoAmbientProbe(class/method).DRY / behavior preservation
The extension contains zero sampler logic.
ProgressProbegained an observer-mode construction path sharing the existing samplers: group id viaSupplier(tracks mid-testGroupOption.NEW_GROUPswitches), multi-topic sampling when unfiltered, watermark maps re-keyedInteger → TopicPartition(removes a latent cross-topic partition-number collision, per the re-key-don't-parallel-cache rule). The chaos construction path is unchanged in signature, thread name, log prefixes, bounds, and cadence - #85's W4 per-scenario toggles (disableRebalanceDwellViolation()/withNoProgressWindow()) carried over intact through the master merge and are orthogonal to observer mode (verified by the automatic review of the final head).Review hardening (post-review rounds)
An 8-reviewer pass plus automatic-review rounds produced fixes, landed in
ffbe9bc5and76ec2ec5(finding-by-finding tables: review response, bot-review dispositions):AfterTestExecutionCallback(before@AfterEachcloses the admin client), withafterEachas an idempotent safety net for the@BeforeEach-throws path (semantics verified against the pinned junit-jupiter-engine bytecode).KafkaClientUtils.close()nulls the volatileadminfield in afinally; samplers use the newadminIfOpen()Optionalaccessor - the open()..close() window is named at the call site, no magic nulls.violate()logs DEBUG in observer mode (violations still recorded for the autopsy); chaos-mode ERROR unchanged.Modeenum (CHAOS/AMBIENT_OBSERVER) replaces null-sentinel mode encoding; chaos constructor gainsrequireNonNullguards;volatile groupIdfor cross-thread supplier reads.AmbientProbeExtensionTest, runs under plainmvn test, no broker):isDisabled()opt-out matrix, all autopsy formatting branches,beforeEachfallbacks,stop()idempotence, observer/chaos log-level contract.docs/inflight.mdshrunk per its scope rule (retired the merged feat(chaos): Chaos Pain Suite - calibrated, portable bug-detector pack (finds the drain-zombie on master blind) #83/feat(chaos): hunt silent commit stalls by revoking partitions mid-work (Phase 2, W4) #85 entries; the ambient-probe roster bullet now points here).Verification (all executed)
ZOMBIE_MEMBER, dwell peak 15.665s, own peaks line unchanged) AND the extension produced its first real autopsy, naming the zombie plus a frozen-partitions table.ChaosRevokeUnderWorkITcompiling against the mergedProgressProbe): 311 run / 0 failures attributable to this PR;AmbientProbeExtensionTest14/14 (re-run 4x, no flakes). Each full-suite run surfaced one pre-existing parallel-execution flake in an unrelated main-suite test (different test each run; all pass solo).40d3d9f): no new issues; conflict resolution audited clean.Implemented by a subagent in an isolated worktree; reviewed, verified, and committed by the orchestrating session.
Checklist
CHANGELOG.adoc, under== Unreleased)AmbientProbeExtensionTest(14 unit tests)🤖 Generated with Claude Code
https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18