Repository navigation
feat(chaos): Chaos Pain Suite - calibrated, portable bug-detector pack (finds the drain-zombie on master blind) - #83
Conversation
…same-repo only) Optional, non-gating fast feedback that runs unit, integration and performance as parallel matrix jobs on a self-hosted many-core runner (label `highcpu`), guarded to same-repo PRs so fork code never executes on it. The GitHub-hosted gate in maven.yml stays the required check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…rkflow # Conflicts: # .github/actionlint.yaml
…-only The high-CPU self-hosted matrix now runs unit + integration + performance + mutation as parallel jobs (a strict superset of the mac runner, which is offline indefinitely). The mac workflow is switched to workflow_dispatch-only so it stops queuing dead jobs on every PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
failingHttpCall drove an HTTP call at a dotless bogus hostname ("xxxxxxxxx") and asserted the failure
was a DNS *resolution* error ("failed resolve"). That only holds where the name fails to resolve - on any
network with a local resolver + search domain (e.g. a Pi-hole LAN) the name resolves and the failure mode
becomes "connection refused", so the test fails there while passing on GitHub's public DNS.
Point the bad request at a closed local port (127.0.0.1:1) instead: an immediate, deterministic
"connection refused" on every environment, no DNS lookup at all (and faster than one). Keep the real
failed==true check; assert the deterministic "connection refused" message. Found bringing up the
self-hosted high-CPU runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…with failingHttpCall) The failingHttpCall fix (127.0.0.1:1 -> connection refused) also flipped this sibling test through the shared getBadRequest() helper, so its assertion is updated to match. Caught by ce-review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Load was ~18/32 with 34GB free - the box is I/O-bound (cold cache), not CPU-bound - so use all the cores. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…ateCommittedOffsetIT Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…epo guard (ce-review P2) The if: guard lives in the PR-head file, so the real RCE backstop is the repo Actions setting "require approval for all outside collaborators". Documented in the workflow header + referenced SELF_HOSTED_RUNNER.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Unit forkCount 1C->2C, Integration 8->16 (probe the oversubscription ceiling). Mutation PIT threads 2->16 (CPU-bound, stays in the matrix). Performance split into its own job that needs the matrix, so the throughput benchmark measures on a quiet box (solo 1m54 vs 7m18 when it shares the box). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…formance Doubling was flat (unit 2C=1C) to worse (integration 16 slower than 8), so back to the knee. Performance returns to the concurrent matrix - it is a correctness gate here; noisy timing is fine and isolation wrecked total wall-clock (it queued behind a 17-min mutation). Mutation keeps threads=16. Accurate benchmarking -> separate on-demand run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Unit/Integration now also enable JUnit method-parallelism (-Dparallel-tests=true) on top of forking, forks reduced so forks x threads ~= cores - tests whether thread-parallelism is still flaky after recent fixes (revert to fork-only if red). ci-mutation-test.sh: on a PR (GITHUB_BASE_REF) mutate ONLY changed core classes, skip if none - the full internal.* sweep has never completed. Full sweep still runs for push/nightly (no base ref). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…eads finding fork×threads (parallel-tests=true) was safe (threading no longer races - unit green; integration red was the known flaky PartitionStateCommittedOffsetIT) but no faster than fork-only, so back to 1C/8. PIT: grep no-match under set -euo pipefail made the changed-class skip exit non-zero (0m19 red); wrap grep with || true so it exits 0. Recorded the thread-parallelism-vs-forking finding in inflight next to the shelved Step-2 question. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…bserve a green board TEMPORARY - not a fix. Disables the known-flaky PartitionStateCommittedOffsetIT.committedOffsetRemoved (awaitility timeout, in inflight, fails on GitHub-hosted too) purely to see a fully-green highcpu run. Revert and harden the test properly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
Now that ci-mutation-test.sh scopes PIT to changed classes on PRs (GITHUB_BASE_REF), it is fast enough for the 2-core hosted runner - so mutation runs on GitHub-hosted as well, not just the self-hosted box. Non-gating (continue-on-error). fetch-depth: 0 so the changed-class diff resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
…already passed Each GitHub-hosted test job, before running its slow 2-core suite, polls whether the self-hosted highcpu equivalent for the same commit already PASSED (ci-concede-check.sh). If so it skips and goes green; the hosted job stays the required gate but finishes faster when grumpy wins. Never depends on the self-hosted runner: offline/queued/slow/failed -> run the tests normally. Bounded wait, continue-on-error so a check bug can never fail the gate. Caveat documented: conceding trusts the self-hosted env (env-specific failures could be missed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
… the green-board experiment) Reverts the temporary experiment disable. The test is still the known flaky awaitility timeout (tracked in inflight) - harden it (adaptive await / underlying timing) rather than leave it disabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
The check name is "<workflow> / <job>", and both halves carried "High-CPU"
plus verbose filler ("Build & Test - ...", "self-hosted, optional"), pushing
past the PR merge-box truncation width.
- workflow name: "PR High-CPU Fast Feedback" -> "highcpu"
- job name: "Build & Test - <suite> (high-CPU self-hosted, optional)"
-> "<suite> (optional)" (keeps the non-gating signal)
Result: "highcpu / Unit (optional)", "highcpu / Integration (optional)", etc.
- longest ~35 chars, well under the cutoff.
Kept the concede matcher in sync (it resolves the highcpu run/jobs by name):
bin/ci-concede-check.sh WORKFLOW default -> "highcpu", JOB_PREFIX -> "${SUITE}"
(the job is now named "<suite> (optional)", so a suite-name prefix still
uniquely matches). Added a keep-in-sync comment on the job. Updated the
SELF_HOSTED_RUNNER.md reference.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ytagqk6daL4uTtbebQNNv
… behind Mutation's slow cancel A single workflow-level concurrency group meant every new push's jobs waited for the ENTIRE previous run to complete before starting. cancel-in-progress signalled the old run immediately, but its Mutation (PIT) maven/JVM tree rides out the runner's full cancellation grace (~5 min) before force-kill - observed on PR #80: push at 22:41:05, old Mutation force-killed 22:46:06, all four new jobs started 22:46:09. Head-of-line blocking on the one job nobody needs fast feedback from. Job-level per-suite groups (highcpu-<suite>-<branch>) mean a new push supersedes each suite independently: Unit/Integration/Performance predecessors are finished or die fast, so they start within seconds; only Mutation queues behind old Mutation, which is fine for a 30-minute advisory job. No change to runner usage. actionlint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…ner' in comments Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…robes (W1 skateboard) Implements Phase 1 of the Chaos Pain Suite design (branch docs/chaos-pain-suite-design; plan under docs/plans/). Additive-only on the uber experiment composition so it transplants onto the rebased-#29 slices as a file copy - zero edits to #29-owned files. - ChaosConductor: seeded, replayable action scheduler (STOP_DRAIN / STOP_NO_DRAIN / RESTART / JOIN_NEW) with EnumMap-deterministic weighted picks, a timestamped action timeline logged every run (the autopsy's first artifact), a join-after-stopDrain bias knob (the zombie-drain collision the calibration targets), and its own per-instance state machine over ManagedPCInstance - including raw closeDrainFirst() for DRAIN stops, which the managed API doesn't expose. - ProgressProbe: independent SLO samplers - progress watermark (30s, with tail slack for legitimately-sleeping heavy records), rebalance-dwell zombie probe (60s: >> seconds-healthy, << 5-min zombie; keyed on protocol-unresponsiveness, NOT partition-holding, which is correct drain behaviour), drain-completion bound (150s, must exceed the heavy tail), plus the end-of-run correctness ledger (no loss ever; duplicates bounded per disturbance, capacity-shaped). - ChaosChurnStormIT (W1): 12->16 fleet, 80 partitions, 100k messages with heavy-tailed work (1-in-4000 records sleeps 90s) - the heavy tail is what stretches drains into the probe's discrimination window, since the zombie-drain defect freezes the group for the DURATION of a drain. - ChaosConductorPlanIT: seed-determinism regression (same seed = same plan). - @tag(chaos) excluded from all default suites (pom default now performance,chaos; comment documents the multi-group override gotcha); chaos-pain.yml workflow_dispatch runs it on-demand on the highcpu runner (seed + reps inputs, artifacts uploaded, never gating). Validated: 3 green storm runs (500k and 2x100k msgs - zero loss, duplicates ~500-2700 vs allowances), determinism 2/2, default-exclusion verified (name-matched chaos test runs 0 tests without -Dincluded.groups=chaos). Calibration against the planted zombie-drain defect arm is the next commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
…(portable bug-detector pack) Completes the suite for standalone landing on master: - ManagedPCInstance extracted from the confluentinc#857 investigation branch (PR #29, credited) - the multi-instance PC lifecycle harness; verified mainline-API-only. When #29 rebases, its harness slice simply drops its copy. - Suite files at calibrated state: non-interruptible heavy tail (PC's close force-interrupts stragglers ~11s, so interruptible sleeps cannot stretch drains into a discriminable window), rebalance-dwell ZOMBIE bound 15s set from same-seed A/B measurement (healthy peak 6.7s vs defect 20.1s), peak signature logging, hotter conductor (0.5-1.5s ticks, join-after-drain bias 0.9). Calibration record (see the design doc on docs/chaos-pain-suite-design and plan on experiment/chaos-pain-suite-phase1): planted-defect arm 3/3 RED via ZOMBIE_MEMBER/REBALANCE_BLOCKED (dwell 15.6-15.9s), all-fixes arm 3/3 GREEN (4.5-8.9s), and PLAIN MASTER: RED via the same probe (15.6s) - this branch's own base carries the drain-zombie bug (fixed on PR #80), and the suite finds it blind. That is the point: a portable detector, excluded from every gating suite (@tag chaos), temporarily mergeable into any PR to expose fix-absent bugs and verify fixes. Run on demand: -Dincluded.groups=chaos [-Dchaos.seed=N], or the chaos-pain.yml workflow_dispatch on the highcpu runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
Dependency ReviewThe following issues were found:
License Issues.github/workflows/chaos-pain.yml
OpenSSF Scorecard
Scanned Files
|
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ✅ jscpd (language-agnostic)
No new clones introduced by this PR. Powered by astubbs/duplicate-code-cross-check |
✅ SpotBugs ReportNo bugs found (new bugs only — baseline from base branch excluded). |
…it; nothing is 'planted'/injected 'Planted' (mutation-testing jargon) wrongly implied synthetic fault insertion. The defect arm is git archaeology: the genuine historical bug in its original code, selected by basing on the pre-fix commit. The only engineered element is the sensitizing workload (non-interruptible heavy tail) that makes the real defect's freeze long enough to discriminate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
… stagnation Detects protocol-INVISIBLE stalls: group STABLE, heartbeats + polls flowing, no rebalance pending (so the broker's 5-min eviction clock never starts), yet a partition's lag is real and its committed offset stops moving - the 'locks forever until manual restart' class from the confluentinc#857 reports, which the Class 1 zombie probe cannot see. Mechanism: every 5th sampler tick, admin listConsumerGroupOffsets vs listOffsets(latest); violation CLASS2_STALL/LAG_STAGNATION when lag >= 50 and the committed offset has been stagnant > 150s. Peak (maxLagStagnation) logged every run. Calibration so far (test-the-test discipline): - First healthy run FIRED at 151s: a real measurement, not a probe bug - a hard stop interrupts a heavy record mid-dwell and at-least-once re-runs it fresh, legitimately blocking that partition ~2 chained dwells. Fix per the calibrate-the-workload lesson: heavy dwell 90s -> 45s (the zombie window only needs drains occupied; close bails at ~11s regardless), so a 2-dwell redelivery chain (~90s) fits under the 150s bound. Arithmetic recorded in both javadocs. - Re-validated: healthy arm 2/2 GREEN (peak lag stagnation ~39s, ~4x margin); defect arm still RED on ZOMBIE_MEMBER with unchanged dwell signature (15.7s vs 15.6-15.9s at the old 90s dwell). - RED calibration of this probe awaits the W4 revoke-under-work trigger scenario (next Phase 2 step). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
… the pom The gating scripts hardcode -Dexcluded.groups (overriding the pom default), so adding 'chaos' only to the pom let ChaosChurnStormIT leak into the required Integration Tests check - where it promptly went RED on this branch's master base (the detector detecting; but it must not gate). Same class of bug the quarantine-lane PR #84's contract test now pins for the 'quarantined' group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
|
Claude finished @astubbs's task in 5m 35s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
This is a well-engineered, heavily self-documented suite — the determinism rework ( 🔴
|
No workflow targets [self-hosted, macOS] anymore - the mac fast-feedback lane was superseded by the grumpy/highcpu workflows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 5m 37s —— View job Claude finished @antony Stubbs's taskTodo list
ReviewReviewed the diff ( The core design is solid and the 1. Shell/script-injection risk in the new
if [ -n "${{ inputs.seed }}" ]; then SEED_ARG="-Dchaos.seed=${{ inputs.seed }}"; fi
env:
CHAOS_SEED: ${{ inputs.seed }}
run: |
...
if [ -n "$CHAOS_SEED" ]; then SEED_ARG="-Dchaos.seed=$CHAOS_SEED"; fi2.
3. Ledger duplicate allowance is very loose at this scenario's scale (informational, already reasoned about)
Nothing else stood out. |
Reply to a review comment in its own thread and mark it resolved when addressed, rather than a top-level PR comment (which leaves the conversation unresolved and blocks merge). Recording the rule that surfaced while handling PR #75 review feedback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Claude finished @astubbs's task in 4m 1s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
Only two commits landed since the last full review round ( 🔴 Still open:
|
Addresses the Claude auto-review findings on PR #83: 1. chaos-pain.yml: workflow_dispatch inputs (seed, reps) now pass through env vars instead of ${{ }} template substitution inside run: - raw interpolation executes attacker-shaped input as shell on the self-hosted runner (the review flagged seed; reps had the same vector). Also ${SEED_ARG:+...} quoting per shellcheck. 2. ManagedPCInstance.run(): try/finally MDC.remove - work-stealing pool threads are reused across instances, so the Runner-N tag leaked into later instances' setup logs between restarts. Finding 3 (loose ledger duplicate allowance) intentionally unchanged: it is the documented vacuous-bound guard; the evidence-based tightening is rostered in docs/inflight.md behind revoke-event instrumentation. Verified: test-compile clean, actionlint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 2m 53s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
Only one commit landed since the last review round: ✅ Both targeted fixes are correct
🔴 Still open:
|
Review response (re: the auto-review) - addressed in c2ce5171. Shell injection in 2. MDC leak in 3. Ledger duplicate allowance - intentionally unchanged, as the review anticipated. It's the documented vacuous-bound guard (record-loss zero-tolerance does the real work). The evidence-based tightening is rostered in Verified before push: 🤖 Generated with Claude Code |
…hecklist gate (#97) Resolutions: newer #97 checklist wording in CHANGELOG/README/AGENTS (keeping this branch's in-thread review-response policy bullet alongside); ci scripts keep the chaos-inclusive exclusion list per the quarantine contract test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 2m 5s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
SummarySince the last full review round ( 🔴 Still open:
|
The PR-checklist gate (now on master via #97) requires it - and the suite is changelog-worthy by house style (CI/dev-facing entries are in). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 1m 8s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
SummaryThe only commit since the last full review round ( 🔴 Still open (unchanged, 4th time flagged):
|
…unds 3-5 follow-up The chaos-pain job deliberately excludes @Quarantined scenarios (decision in c3ba4e1: the known #80-pending W1 RED must not drown the tripwire signal), which currently means it selects ZERO tests. The review's two outstanding asks, now addressed: - chaos-pain.yml timing summary: prints a loud 'ZERO chaos tests selected - this run measured NOTHING' banner (instead of an empty table impersonating a real GREEN) when no failsafe reports exist; header comment documents the quarantine interaction and the interim window. - AGENTS.md chaos section + docs/inflight.md: the CI job is no longer described as equivalent to the local full recipe - the quarantine exclusion and the temporary zero-test state are called out explicitly. Coverage returns when the W4 variants (#85/#87) or the #80 fix land. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 1m 3s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
SummaryOne commit landed since the last review round: ✅ The zero-test-selection finding (flagged in rounds 1, 3, 4, 5) is now resolved
This is exactly the second of the two options I'd suggested each round ("keep the exclusion deliberately and add the doc caveat + a visible zero-tests signal"), executed cleanly across all three of the places that needed it. Remaining nit (non-blocking, cosmetic): Everything elseNo new code changes since the last round (only No outstanding blocking issues from this review. |
Review response (re: the recurring zero-tests finding, rounds 3-5) - addressed in 816e77cThe decision stands as the interim tradeoff from c3ba4e1:
Coverage returns automatically when the W4 variants (#85/#87) or the #80 fix land - either adds non-quarantined chaos scenarios back into the job's selection. Earlier rounds' fixes (workflow input injection via 🤖 Generated with Claude Code |
'Temporarily merge the chaos branch into the PR under test' only made sense while the suite lived on an unmerged branch. Once this PR lands the suite is on master, so the recipe is: on the fix PR's branch (merge master in if it predates the suite), run pre-fix vs fix commits and watch RED->GREEN. Fixed in AGENTS.md + the W1 class javadoc; the PR body is updated in the same pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 50s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
SummaryOne commit landed since the last full review round: The fix corrects a recipe that no longer made sense once this PR is destined to land on Verified:
No functional, security, or test-coverage changes in this commit, and no new issues found. Standing item (informational only, not re-litigated)The |
…s dispatch-only Review of today's runner meltdown: independent self-hosted workflows can't see each other's load, and the per-PR trigger semantics (cancel superseded) fought chaos-pain's dispatch semantics (queue, never kill). Split cleanly: - bin/chaos-test.sh: shared runner - the chaos mvn invocation, timing summary and the zero-test quarantine banner (emitted on EVERY exit via trap, so a RED run still gets its autopsy summary). Inputs via env only. - pr-highcpu-fast-feedback.yml: new 'Chaos Pain Suite' matrix entry - the per-PR ambient tripwire, sharing the lane's same-repo guard and per-suite superseding. NOT optional (continue-on-error false, bare check name): a chaos RED is a real finding and shows red. - chaos-pain.yml: workflow_dispatch only (seed/reps hunts), queue-never- kill preserved; delegates to bin/chaos-test.sh. - AGENTS.md / docs/inflight.md / CHANGELOG+README bullet updated. Verified: bash -n, actionlint both workflows, copyright scan clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 4m 28s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
Only one commit landed since the last review round: 🔴
|
… reported green Review round 6 caught a real regression from effa942: an EXIT trap's own last command becomes the script's exit status, and everything in emit_summaries is made to succeed - so bin/chaos-test.sh exited 0 even when mvnw verify failed, turning a real chaos RED green at both call sites (the non-optional highcpu lane entry included). Exactly the silent-success pattern five prior rounds stamped out. Fix: capture $? first, re-exit with it after emitting summaries. Verified with stub harness: failing run propagates exit 1 (banner still emitted), passing run exits 0. Also: CHANGELOG/README no longer describe the highcpu lane as blanket continue-on-error - the Chaos Pain Suite entry is deliberately not (round-6 minor note). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
Review response (round 6) - trap bug fixed in b35d46bThe EXIT-trap exit-code swallow: confirmed real, fixed. Textbook version of exactly what the review said - the trap's own always-succeeding last command became the script's exit status, so a genuine chaos RED reported green at both call sites. Minor note also taken: CHANGELOG/README no longer describe the lane as blanket The offline-runner tradeoff flag: consciously weighed, keeping as-is - a pending/red chaos check from an offline runner is acceptable noise for an unmissable tripwire, and it stays out of the required set precisely so it can never block a merge. 🤖 Generated with Claude Code |
|
Claude finished @astubbs's task in 1m 25s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
SummaryOne commit landed since the last review round: ✅ The EXIT-trap bug is correctly fixed
emit_summaries() {
local ec=$?
summary || true
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then summary >> "$GITHUB_STEP_SUMMARY" || true; fi
exit "$ec"
}
trap emit_summaries EXITThis is the correct idiom: I traced the logic by hand rather than executing it live (sandboxed shell execution in this environment required approvals I didn't have for this pass), but the pattern matches the well-established fix for this exact bash gotcha and the code reads correctly against both the failing and passing paths. The accompanying No new issues foundNothing else changed in this commit. All previously-reviewed code and CI wiring remain unchanged since the last clean passes. This closes out the last open finding from the review series — no outstanding blocking issues. |
…work - head start on #83 landing Non-trivial resolutions (base evolved past what this branch extracted): - ChaosScenarioBase helpers upgraded to base's new W1 epilogue semantics: settleFleet waits out stopAsync close-pending, settleRun takes/kills the pcExecutor, assertScenarioSlos uses the full replay command in messages, adds the unclassified-failure-cause canary sweep, and uses conductor.getDisturbanceCount() - W1 keeps its slim helper-calling body + base's @Quarantined + replayCmd - W4 adopts the same replayCmd pattern for message parity - W4 plan tests ported from removed planActions to planTicks/TickDraws (base's single-draw-path rework); ChaosConductorPlanIT 4/4 green - ManagedPCInstance: extraConsumerProps merge re-seated in base's restructured run(); Config field unchanged - AGENTS.md: dropped this branch's chaos signpost bullet (base now has a full Chaos Pain Suite section); added W4 + ChaosScenarioBase to it - inflight.md: chaos Phase 2+ roster kept, updated to post-merge reality (cooperative lever -> PR #87, ambient probe -> PR #86); dropped the fork-header roster entry (base's copyright work already flipped them) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018igSVt74wPQAbRNHkXS4R8
…dencies' Two fixes for the newly-required check: - synchronize added to the trigger types: required checks are evaluated per head SHA, and without it a plain push produced no run - the check sat 'Expected - waiting for status' forever, blocking merges. - Job renamed 'Check Dependencies' -> 'Check PR Dependencies' (clearer: it gates stacked-PR 'depends on #N' ordering, not library dependencies). Ruleset context updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ALhTf1TRA2S6Ue5rkiQdK
|
Claude finished @astubbs's task in 42s —— View job Code review — PR #83 (Chaos Pain Suite)Todo list
SummaryOne commit landed since the last review round: ✅ Clean, well-justified fix on:
pull_request:
- types: [opened, edited, closed, reopened]
+ types: [opened, edited, closed, reopened, synchronize]This closes a real GitHub Actions gotcha: once a workflow is configured as a required status check, required checks are evaluated per head SHA. Without The accompanying rename ( No functional issues found. This is a workflow file, so I can't push a fix myself even if one were needed — not applicable here since there's nothing to fix. Everything elseUnchanged since the last clean pass (round 7, |
…s pre-diagnosed (#86) AmbientProbeExtension (JUnit 5, on BrokerIntegrationTest - all ~104 broker ITs inherit it) runs the chaos suite's ProgressProbe in a new observer mode under every test: rebalance-dwell + per-partition committed-offset lag stagnation, sampled purely from admin-client reads. On test failure it logs an autopsy block (violations, peaks, frozen-partitions table) so a generic Awaitility timeout arrives pre-diagnosed: "zombie member blocked rebalance 15s" vs "partition 21 committed frozen lag=1132" vs "probe clean - fault is likely in the test itself". Observer mode never fails a test; silent on green (DEBUG one-liner). Escape hatches: -Dambient.probe=off, @NoAmbientProbe. ProgressProbe: explicit Mode enum (CHAOS / AMBIENT_OBSERVER), group id via Supplier (follows NEW_GROUP switches), watermark maps re-keyed Integer -> TopicPartition (fixes a latent cross-topic collision), observer violations recorded but logged DEBUG. Chaos-mode behavior unchanged; #85's per-scenario toggles carried through intact. Lifecycle hardening: sampler stops in AfterTestExecutionCallback (before @AfterEach closes the admin client; verified against the pinned junit-jupiter-engine bytecode), afterEach as idempotent safety net; KafkaClientUtils.close() nulls the volatile admin field in a finally, samplers use the lifecycle-safe adminIfOpen() Optional accessor. Tests: AmbientProbeExtensionTest (14 broker-free unit tests) - isDisabled() opt-out matrix, autopsy formatting branches, beforeEach fallbacks, stop() idempotence, observer/chaos log-level contract. Docs: AGENTS.md points test-debuggers at the autopsy first; docs/inflight.md shrunk per its scope rule (#83/#85 entries retired). Verified: real autopsy produced on a W1 chaos failure with chaos probe behavior bit-identical; zero ambient output on green runs; unit suite 311/0 attributable; clean automatic review of the final head.
1 participant
depends on #75
A portable bug-detector pack: a seeded, calibrated chaos suite that hunts PC's "alive but not progressing" bug class on purpose - excluded from every gating suite, runnable on demand against any branch, and able to probe any fix PR: run it at a commit before the fix (RED, with the violation naming the mechanism) and at the fix (GREEN).
Lands independently of the fix PRs - deliberately early in the merge queue so every subsequent PR can use it.
What
ChaosConductor- seeded, replayable chaos scheduler (-Dchaos.seed=Nreplays a schedule; seed + a timestamped action timeline logged every run). Actions: drain-mode stops, hard stops, restarts, joins - with a join-after-stopDrain bias targeting the zombie-drain collision. The draw stream is a pure function of (seed, tick index) - deterministic replay survives drain-completion timing (review-hardening rework,a52fd9e).ProgressProbe- SLO samplers with violations that name the mechanism: progress watermark (30s), rebalance-dwell zombie probe (15s - see calibration), drain-completion bound, Class 2 lag-stagnation probe (protocol-invisible stalls: group STABLE, heartbeats + polls flowing, yet a partition's committed offset stops moving with real lag -CLASS2_STALL/LAG_STAGNATION, 150s bound), end-of-run correctness ledger (no record ever lost; duplicates bounded per disturbance). Peak signatures (maxRebalanceDwell/maxDrainDuration/maxLagStagnation) logged every run.PROBE_DEGRADEDguards against false-GREEN when sampling itself fails.ChaosChurnStormIT(scenario W1) - 12-16 PC fleet, 80 partitions, 100k msgs with a non-interruptible heavy tail (1-in-4000 records dwells 45s - PC's close force-interrupts stragglers at ~11s, so polite sleeps cannot stretch drains into a probe-discriminable window; 45s not 90s because a hard stop can interrupt a heavy record mid-dwell and at-least-once re-runs it fresh, legitimately blocking that partition ~2 chained dwells - measured 151s at 90s dwell, which the Class 2 probe correctly flagged; 2x45s=90s fits under its 150s bound).ManagedPCInstance- the multi-instance PC lifecycle harness, extracted (with credit) from the Paused consumption across multiple consumers confluentinc/parallel-consumer#857 investigation branch (PR fix(core) confluentinc#857: the revoke path declines the commit lock instead of blocking on it #29); verified mainline-API-only. fix(core) confluentinc#857: the revoke path declines the commit lock instead of blocking on it #29's eventual rebase drops its copy.@Quarantined(owner PR fix(core): draining PC stops polling - 10kHz busy-spin + zombie partition hold (#857-family silent stall) #80, the drain-zombie fix - the registry forces fix(core): draining PC stops polling - 10kHz busy-spin + zombie partition hold (#857-family silent stall) #80 to delete the quarantine on merge). It stays out of the gating suites and thechaos-painjob, and runs in the Quarantine Lane.chaos-pain.ymlruns the suite on-demand (workflow_dispatch, seed/reps inputs) and on every same-repo PR commit on the highcpu self-hosted runner - never gating, per-run timing summary, dispatch inputs injection-hardened viaenv:. It excludes@Quarantinedscenarios, so until the W4 variants (feat(chaos): hunt silent commit stalls by revoking partitions mid-work (Phase 2, W4) #85/feat(chaos): cooperative-sticky variant of the W4 revoke-under-work scenario - PC's first cooperative-assignor coverage #87) or fix(core): draining PC stops polling - 10kHz busy-spin + zombie partition hold (#857-family silent stall) #80 land it selects zero tests - flagged loudly in the job summary.ChaosConductorPlanITpins the seed-replay determinism contract on the real draw path;ProgressProbeLedgerITcovers the ledger boundary arithmetic.Calibration - the suite is a measured instrument, not decoration
Test-the-test, executed against the real historical bug (no synthetic fault injection - the defect arm is simply a pre-fix commit):
ZOMBIE_MEMBER/REBALANCE_BLOCKEDClass 2 probe calibration (added after): healthy arm 2/2 GREEN with peak lag stagnation ~39s vs the 150s bound (~4x margin); its first run fired at 151s on a healthy arm - a real measurement of legitimate redelivery chaining, fixed by calibrating the workload (heavy dwell 90s→45s), not by loosening the probe. RED calibration of the Class 2 probe continued in the W4 scenario family (#85/#87).
Yes: the suite reds on master today, on first contact - that is the point. Master carries the drain-zombie bug (fixed on PR #80); the detector found it blind. It gates nothing, so this PR's own CI is unaffected (the known RED is quarantined under #80, not hidden).
The 15s threshold is empirical, not guessed: identical seeded schedule run on defect and fixed arms, peaks measured (20.1s vs 6.7s), bound set in the gap - arithmetic recorded in the constant's javadoc.
Usage patterns
-Dincluded.groups=chaos -Dexcluded.groups=at a commit before the fix and at the fix - watch RED→GREEN. The empty exclusion deliberately includes@Quarantinedscenarios, so the known-RED detector still fires locally.-Dchaos.seed=Nreproduces the schedule.chaos-pain.yml- on-demand dispatch (seed/reps) + per same-repo PR commit, non-gating.Known coverage + next
Covers Class 1 stalls (protocol-visible rebalance freezes) with a calibrated RED trigger, plus the Class 2 probe (per-partition lag stagnation, protocol-invisible stalls) armed and GREEN-calibrated. Phase 2 landed on the stack: the W4 revoke-under-work scenario (#85) and its cooperative-sticky variant (#87, PC's first cooperative-mode exercise); later ideas (per-cause duplicate-ledger attribution, EoS variants, ambient probe #86) rostered in
docs/inflight.md. Design doc: branchdocs/chaos-pain-suite-design.🤖 Generated with Claude Code
https://claude.ai/code/session_01VbtaCN1Re14pJ99VFW4R18
Checklist
CHANGELOG.adoc, under== Unreleased) - Chaos Pain Suite entry (6f18e8d)docs/inflight.mdroster,docs/QUARANTINED_TESTS.mdregistryChaosConductorPlanITdraw-path determinism,ProgressProbeLedgerITboundary math, quarantine contract/script tests)chaos-pain.ymlpull_requesttrigger is same-repo-guarded (fork code never reaches the runner);workflow_dispatchinputs pass throughenv:(injection-hardened per the auto-review)