Repository navigation
build(streams) astubbs#255: patch Kafka Streams at build time, with Kafka's own suite as the oracle - #379
Draft
astubbs wants to merge 8 commits into
Draft
build(streams) astubbs#255: patch Kafka Streams at build time, with Kafka's own suite as the oracle#379astubbs wants to merge 8 commits into
astubbs wants to merge 8 commits into
Conversation
…hout vendoring them Adds parallel-consumer-streams as an unpublished, alpha leaf module carrying one thing: a reproducible way to change Apache Kafka Streams' package-private internals, and an oracle that proves the change did not alter Kafka's own behaviour. The mechanism. The named classes are unpacked from the published kafka-streams SOURCES jar at generate-sources - twice, a pristine baseline and a working copy - the tracked patch is applied at process-sources, and the working copy is compiled into this module's own target/classes, which precedes the kafka-streams jar on the classpath. Same package, same classloader, so package-private access works without a fork. Only the patch is tracked: no Apache Kafka source enters this repository, and the patch's line count is the reviewable answer to "how little had to change". The oracle. Kafka publishes its compiled tests to Central, so the module takes them as a test-classifier dependency and points a dedicated surefire execution at StreamTaskTest, RecordCollectorTest and ProcessorContextImplTest - which therefore exercise OUR patched classes. Nothing is excluded, relaxed or recompiled, and it runs in the normal test phase on every build. Counts are read from target/surefire-reports-kafka-upstream/ and must be RE-DERIVED, never copied: scoping the run with -Dtest= silently overrides the includes, the suite does not run at all, and the build still goes green having computed nothing. What this patch changes, and why it stops there. AbstractProcessorContext's current record context and current node move to thread-confined slots; ProcessorContextImpl moves to the accessors, because a subclass doing getfield/putfield on those inherited fields defeats the confinement with no compile error to say so; RecordCollectorImpl's two maps become concurrent, because they are written from the producer's I/O thread for every in-flight send. Kafka's own InternalMockProcessorContext needs the same accessor conversion through a second patch, or RecordCollectorTest cannot construct its subject at all. StreamTask is deliberately NOT patched, and there is no Parallel Consumer dispatcher, no switch and no record going through PC. The seam is the next rung of this stack. ShadowedClassLoadingTest asserts StreamTask still loads from the jar, which is the tightest available statement that the generated set is exactly what the pom declares - and it is what will have to change, visibly, when the seam arrives. Publication is off - maven.deploy.skip, gpg.skip and the publishing plugin's skipPublishing - and that is the settled position rather than an oversight: merging an isolated leaf module is cheap to reverse, publishing an artifact carrying classes in Apache's own package namespace is not. Classpath order is a convention rather than a guarantee, class loading is per class so the result is always a mixture of two versions, and JPMS forbids split packages outright. maven.install.skip is deliberately NOT set, so a local install still works for experimentation. NOTICE carries the Apache 2.0 section 4(b) statement naming the modified classes. Two things cutting this fresh from master surfaced that its origin branch predates. The enforcer's RequireUpperBoundDeps fails on jackson-annotations once jackson-databind is promoted ahead of what kafka-streams resolves, so annotations is declared at the same version. And forbiddenapis fires on locale-dependent String.format calls in Kafka's RELEASED source, at sites the patch does not touch; the generated tree is excluded rather than patched, on the same reasoning that already scopes ArchUnit here - fixing them would grow the change-set this design exists to keep small. Anything of ours landing later sits under bz.stub and is not covered by that exclusion. Verified: full reactor clean install green; the module's own tests and Kafka's suite green with counts read from the report XML; and the empty-patch control arm re-run, leaving the generated tree byte-for-byte identical to the released sources. Copied from #271, which keeps the execution seam and its measurements.
…ate drifting from its self-test Three gate changes, all caused by a fifth library module arriving. The copyright scanner reports any file it cannot classify, which is the design - an extension list goes stale silently the moment a new file type appears, whereas an unclassified file fails the build. A tracked `.patch` is the new type. It is exempt rather than enforced: its content is a diff of third-party source, it is regenerated wholesale by its own tool so a header would not survive, and the attribution obligation it carries is discharged by NOTICE naming the classes it modifies, not by a per-file notice. bin/check-test-log-config.sh gains parallel-consumer-streams. A library module whose logback-test.xml nobody guards is exactly the drift the gate was written for - core had reached root=info that way - and the module's own config was written to the gate rather than copied from a stale one. The self-test's REGRESSION is the interesting part. It restated the module list as a second copy and built its fixture tree from that copy, so adding a module to the gate did not merely go unchecked - it INVERTED the self-test: the fixture omitted the new module, the gate correctly reported it missing, and every "expected pass" case then reported a gate bug that did not exist. Four cases went red for a gate that was working. The self-test now reads the list out of the gate with an anchored parse and fails loudly if it reads nothing, so the two cannot disagree again. A read loop rather than mapfile, because macOS ships bash 3.2 and mapfile fails there as "command not found" on a line nobody looks at twice. Verified: bin/check-all.sh green, and both self-tests (copyright scanner, log config) pass.
…he module's arrival makes false The technique write-up is the durable half of this PR. Build-time patching plus classpath shadowing fails SILENTLY when it is wrong - if the jar's class wins the race, every test still passes while measuring the unmodified library - so the write-up leads with the practices that exist only because the mechanism gives no natural signal: prove the shadowing directly rather than inferring it from behaviour, keep the empty patch runnable as a control arm, run the library's own suite against the patched classes, and always clean, because unpack preserves archive timestamps and the compiler then decides nothing needs rebuilding. It also carries the two rules for reducing a patch safely - the hunk count is a proxy that is wrong in both directions, and the real invariant is that every added and removed line survives - which is what this PR relied on to cut the seam out of an existing patch. Three records assert things the module's arrival makes false, and the correction in each case is the same distinction: a directory in the reactor is not an artifact anyone can depend on. - The workstream signpost said none of this work was on master. Now the machinery is, and nothing else - so the note says which half, and how the rest arrives (a stack reconstructed from master by copy, with the branch forest kept as the evidence record rather than merged). - The held feature records and the staged maturity row both said "whichever PR lands each module". Read literally, this PR is that trigger, and moving them would assert an opt_in of "artifact dependency" for an artifact that publishes nothing. Both now say publishable rather than landed, and the streams row carries the worked example of why, at the point somebody would otherwise move it. AGENTS.md's module table gains the row, marked unpublished, pointing at the module README.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF ScorecardScorecard details
Scanned Files
|
5 tasks done
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ✅ jscpd (language-agnostic)
No new clones introduced by this PR. Powered by astubbs/duplicate-code-cross-check |
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
|
…eadlock, with its seed Seen on this branch's first CI run, in the Chaos lane: ChaosRevokeUnderWorkCooperativeDrainIT, seed 1159274055608047297. Same discriminator the captures above were written to distinguish - the poll thread BLOCKED acquiring a monitor held by pc-control inside commitOffsetsThatAreReady, reached from onPartitionsRevoked - so it is the standing lock-ordering line, not a new cause. Written down because the log expires and the seed is the asset, which is what that file asks for; nobody has replayed this seed either. Not this branch's defect: it adds an unpublished Kafka Streams build-machinery module and changes no core Java, no locking and no rebalance path. What it brought in is master.
…fter the merge bin/check-branch-self-reference.sh flags a line naming the branch or PR being written on, because after the merge the branch is gone and the sentence has to still mean something. The attribution paragraph is reworded so it reads as a record rather than as news, and carries the checked-begin/checked-end markers the rest of that file already uses.
astubbs
marked this pull request as draft
August 31, 2026 02:51
… produced The Chaos lane ran twice on the same branch, separated only by two documentation commits - no Java, no pom, no workflow. First run red on the BLOCKED-on-monitor discriminator, second run green: 1 of 2, different seeds, on a tree that is identical for every purpose this failure could care about. That is the closest-together same-tree pass/fail pair this file has, and it says what the looser pair already cited says - the trigger is the schedule, not the tree. Recorded as intermittency, NOT as a clearance: a green run has never been read as one here, and the seed above is still unreplayed.
The streams module gets Kafka's `sources` and `test-sources` classifier jars through two `dependency:unpack` executions that name them as `<artifactItems>`. `prepare-deps` warms the cache with `dependency:go-offline`, which resolves the declared dependency graph - and an artifactItem is not in that graph, nor is either jar declared anywhere else. So both stayed cold, and every lane that built this module fetched them from Maven Central inside `generate-sources`. That put them on the CDN route lottery recorded in docs/solutions/build-errors/maven-central-timeout-azure-west-regions-2026-04-21.md: an exactly-240s read timeout, and re-running does not reliably help because the runner is often reassigned to the same region. It struck #394 and #395 - Unit and Integration both red at `unpack (unpack-kafka-streams-sources)` with zero tests run, so the lane named "Unit Tests" pointed at a subsystem that never reached compilation. Same plugin-resolves-for-itself class the proxy stack already warms three times over (protoc, scala-maven-plugin, the SpotBugs detector plugins), reached from a new direction: here the plugin resolving for itself is maven-dependency-plugin. The step follows that established pattern - version derived from the pom property rather than a literal, a non-version help:evaluate answer is a hard error naming the property, and the jars are asserted on disk afterwards. Rejected alternative: warming by building the module, which is what the Connect spike branch does. It works, but costs a full build of core inside this job, and that branch records why it cannot be shortened to a cheap phase-only walk - the module's test-scope resolution wants core's tests-classifier jar, which does not exist until core reaches `package`. Two jars are the whole gap and `kafka.version` is a real root-pom property, so naming them is cheaper and no more prone to rot. Verified by extracting the step's script from the workflow and running it: - Empty local repository, script otherwise verbatim: exit 0, and both jars arrive at exactly the paths the assertions name, byte-identical to the populated repository's copies. - Property renamed to `kafka.verzion`, one term changed and nothing else: exit 1, `::error::kafka.verzion did not resolve to a version from the root pom (got 'null object or invalid expression')`, and nothing fetched. The guard fires and names the property. - On-disk assertion is independently load-bearing: a run where the `get` goal succeeded but the asserted path was elsewhere exited 1 on the `test -s`, which is the case the exit code alone would have called a pass. - Sufficiency: after the warm, `-pl parallel-consumer-streams -am generate-test-sources` runs OFFLINE from a cleaned target and both unpack executions succeed, producing the unpacked sources. Offline makes any Central fetch a hard failure, so this shows the warmed jars satisfy the artifactItems resolve rather than merely sitting near it. `kafka.version` is the default every live lane builds at. The one job that overrides it, `test-kafka-compat`, is `if: false`; the comment says so, so re-enabling it comes with the note that its Kafka version falls outside this warm.
… owns the mechanism The note was written on feats/ks-streams-task-lifecycle, two rungs above the branch that introduced the gap - the same misplacement the flake note on this stack was already moved for. The unwarmed workflow and the module's artifactItems both live here, so the note describing them does too. Shrunk to what is still open, because the defect itself is now fixed. The mechanism - why go-offline does not resolve an artifactItem, why the warm names coordinates instead of building the module, what the two guards catch - has a durable owner in the warm step's own comment, so the note points at it rather than keeping a second copy that drifts. What remains open, and is the reason this is not a `git rm`: - Every branch carrying parallel-consumer-streams without the step is still exposed, until it merges forward. The note gives the two commands that list them rather than a list that is wrong within a day. - feats/ks-streams-task-lifecycle still has the pre-fix copy of this file and already contains this branch's earlier commits, so merging forward collides add/add on this path. The note says which side to take. - test-kafka-compat is the one job whose Kafka version falls outside the warm. It is `if: false`, so it costs nothing today and re-opens this if re-enabled.
astubbs
added a commit
that referenced
this pull request
Sep 1, 2026
…chinery rung Merges `feats/ks-streams-fork-machinery` (#379) into this rung, so the whole stack above it stops paying the Maven Central coin flip. Two commits arrive, and only one of them is code: - `ec432b439` teaches `prepare-deps` to warm Kafka's `sources` and `test-sources` classifier jars. This module fetches them through explicit `<artifactItems>`, which `dependency:go-offline` does not resolve, so every lane that built the module downloaded them live from Central inside `generate-sources` - the exact phase the region-dependent 240s read timeout bites. It struck #394 and #395 with Unit and Integration both red at `unpack (unpack-kafka-streams-sources)` and zero tests run. - `311f0b797` moves the ledger note for that gap onto the branch that owns the mechanism. Ordinary merge, no history rewritten. The merge was clean - the workflow hunk and the note are both additions this branch did not have. `bin/check-all.sh` exits 0 in this worktree before and after, all 15 runnable gates passing, so the merge is not-worse by measurement rather than by assumption. The repo's pre-commit gate evaluates a different worktree (#382), so this commit uses `--no-verify`.
astubbs
added a commit
that referenced
this pull request
Sep 1, 2026
…ghtings down the stack Merges `feats/ks-streams-execution-seam` (#388), which has just taken #379's post-cut work, into this rung. Three commits arrive: the `prepare-deps` warm for the Kafka `sources` and `test-sources` classifier jars, its ledger note, and the seam rung's own carry of the invalid-timestamps flake ledger. One real conflict, resolved as a UNION rather than to a side. `docs/inflight/test-streamthreadtest-invalid-timestamps-flake.md` exists on both branches - this rung carried it forward with its own seam-off sighting and its CI sighting, and the seam rung then added the two the example rung reported (#391). Diffing the merged file against both parents shows exactly one delta on this side, the example rung's paragraph, and zero on the other: nothing either branch wrote about this flake was dropped. A sighting ledger loses its whole value when a merge picks a side. `bin/check-all.sh` exits 0 in this worktree before and after, 15 runnable gates passing. The repo's pre-commit gate evaluates a different worktree (#382), so this commit uses `--no-verify`.
astubbs
added a commit
that referenced
this pull request
Sep 1, 2026
…cycle rung Merges `feats/ks-streams-refusal-envelope` (#389), which has just taken the two rungs below it, into this one. This closes the whole spine's Maven Central coin flip: every lane that builds `parallel-consumer-streams` from here up now finds Kafka's `sources` and `test-sources` classifier jars in the warmed cache instead of fetching them live inside `generate-sources`. One real conflict, and the incoming file told this branch how to resolve it. `docs/inflight/ci-streams-classifier-artifacts-escape-the-cache-warming-job.md` exists on both sides: this branch wrote it when the defect was open and it still calls the warm a *candidate* fix, while #379 rewrote it once the fix landed - and left an instruction in its own `post-merge` block naming this branch, this path, and which version to take. The post-fix version wins, as instructed. Nothing is lost by taking a side here: the superseded half describes the same defect as unfixed, and the workflow step's own comment is now the durable owner of the reasoning. `docs/inflight/test-streamthreadtest-invalid-timestamps-flake.md` auto-merged, gaining the example rung's two CI sightings; the diff against this branch's pre-merge version is that paragraph and nothing else. `bin/check-all.sh` exits 0 in this worktree before and after, 15 runnable gates passing. The repo's pre-commit gate evaluates a different worktree (#382), so this commit uses `--no-verify`.
astubbs
added a commit
that referenced
this pull request
Sep 1, 2026
…ngs, and what those runs prove The Unit lane of #388 and of #389 both failed on `StreamThreadTest.shouldLogAndRecordSkippedRecordsForInvalidTimestamps[3]` in the first CI run after #379's classifier cache-warm was cascaded into them - same parameterisation, same assertion, every other lane green on both. Recorded for a second reason, which is worth more than the tally: **neither run hit the Maven Central classifier timeout**. That failure had been taking whole lanes down with zero tests run, on these exact branches, and the cascade was done to close it. The suite ran on both, so the warm works, and this flake is now the only thing reddening that lane.
astubbs
added a commit
that referenced
this pull request
Sep 8, 2026
…r-surfacing rung Merges `feats/ks-streams-task-lifecycle` (#394), this PR's BASE, which has just taken #379's post-cut work. Six commits arrive; the only code among them is the `prepare-deps` warm for Kafka's `sources` and `test-sources` classifier jars, which closes this rung's share of the Maven Central coin flip - the failure this branch spent four commits recording sightings of. **This merges the base rung, NOT master, and that is a correction rather than a shortcut.** The task was framed as "merge origin/master so #395 stops reading CONFLICTING", but this PR's base is `feats/ks-streams-task-lifecycle`, not master - so GitHub's CONFLICTING is computed against that rung, and master cannot clear it. The evidence is direct: the sole conflicting path, `docs/inflight/ci-streams-classifier-artifacts-escape-the-cache-warming-job.md`, **does not exist on master at all**, so a master merge leaves the collision untouched while putting this rung 75 commits ahead of its own base and inflating a stacked PR's diff with work no rung below it carries. The whole spine still sits on `b2e6c190d`; changes travel DOWN it by merging the rung below, which is what the three merges arriving here did. One real conflict, and the incoming file left the instruction for it. - `docs/inflight/ci-streams-classifier-artifacts-escape-the-cache-warming-job.md` - resolved to the incoming POST-FIX version's structure, as #379 instructed in the note's own `post-merge` block: take the version that names the warm step, not the one that still calls the fix a candidate. Taken as a straight side, though, that instruction would have dropped four commits of evidence it was never written about: it names the *lifecycle* rung, whose copy was the pre-fix one unchanged, while THIS rung had since added its own sightings to it. So the branch's findings are kept, condensed into a new section written in post-merge terms and marked `post-merge: checked` - the second independent sighting that made this master-state rather than one PR's problem, that Unit and Integration fail together because they share the runner's route to Central, that a re-run is a coin flip rather than a fix, and the markdown-only control arm that settles the attribution. All three remain true for any branch that has not yet merged the warm forward, which is exactly the population the post-fix version's "What is still open" section addresses. Nothing either side wrote and still holds was dropped; what went is the superseded framing of the defect as unfixed, whose reasoning the workflow step's own comment now owns. One sentence was moved to the past tense rather than carried verbatim: the incoming copy says `feats/ks-streams-task-lifecycle` still holds the pre-fix file and will collide add/add. Both rungs that hit that collision have now resolved it as instructed, so as written it was a claim about NOW that this merge falsifies - the exact rot `bin/check-branch-self-reference.sh` exists to catch. The instruction is kept for any rung above these two that has not yet merged forward. `.github/workflows/maven.yml` and `docs/inflight/test-streamthreadtest-invalid-timestamps-flake.md` auto-merged as pure incoming additions - the warm step, and the example rung's two CI sightings. No decision this branch recorded is reversed by the incoming side. Verification. The merge changes **no Java and no pom** - `git diff --stat HEAD` against the pre-merge tip is the workflow file and two markdown notes - so no module's compilation or test outcome can move, and none was re-run on that basis rather than on assumption. `bin/check-all.sh`: 15 ran, 13 passed, 2 failed, and **both failures were reproduced on the pre-merge tip 228cebc in a detached worktree**, so neither is this merge's: - `check-file-refs.sh` - `docs/BUG_857_INVESTIGATION.md`, cited by three `docs/solutions/` write-ups, does not resolve on this branch. Identical before and after. - `check-quarantine-owners.sh` - `ProducerManagerTest.producedRecordsCantBeInTransactionWithoutItsOffsetDirect` names owner PR #262, which has merged since this branch was cut while the quarantine stayed. Live GitHub state, not tree state. `check-branch-self-reference.sh` and `check-issue-refs.sh` both pass, including the new section's `post-merge: checked` block. The repo's pre-commit gate evaluates a different worktree (#382), so this commit uses `--no-verify`; the gates above were run in this worktree instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018rEzrWYFr6oEzy6porczd3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serves #255.
Description
The claim this PR makes, and nothing more: there is a reproducible mechanism for patching Apache
Kafka Streams, and the unmodified path still behaves like upstream.
Kafka Streams'
processor.internalsis package-private, explicitly not an API, and offers no seamwhere Parallel Consumer needs one. This adds
parallel-consumer-streamsas an unpublished alphaleaf module carrying the build machinery for changing those internals without vendoring or forking
them, plus the oracle that proves the change is behaviour-preserving.
There is no Parallel Consumer execution seam here.
StreamTaskis not patched, there is nodispatcher, no switch, and no record goes through PC. That is the next rung of this stack.
How it works
kafka-streamssources jar atgenerate-sources- twice, a pristine baseline and a working copy.process-sources, dry-running first so a rejected hunk fails thebuild rather than leaving a half-applied tree that compiles.
target/classes.siblings still load from it. Same package, same classloader, so package-private access works.
No Apache Kafka source is committed. Only the patch is tracked, and its line count is the
reviewable answer to "how little had to change".
What is in the patch, and why it stops there
The judgement call was the smallest patch that honestly exercises the machinery. A marker-sized
patch would have made the oracle vacuous and left the fixture patch pointless; the full patch from
#271 would have dragged the execution seam in. The seam turned out to be
cleanly separable: only the
StreamTaskhunks reference Parallel Consumer classes. So thiscarries the other three files, whose changes are the concurrency-safety groundwork and are pure
Kafka edits:
AbstractProcessorContextProcessorContextImplgetfield/putfieldon those fields, so leaving it alone defeats the confinement with no compile error to say soRecordCollectorImplHashMaps becomeConcurrentHashMapsKafka's own
InternalMockProcessorContextneeds the same accessor conversion through a secondpatch, or
RecordCollectorTestcannot construct its subject at all - so both patch streams aregenuinely exercised.
The patch was not carved by hand. It was produced by the prescribed procedure: apply the source
patch through
process-sources, deleteStreamTaskfrom both generated trees, runbin/regen-patch.shwith no maven run in between.regen-patch.sh's hunk-count tripwire fired asdesigned, and the real invariant was then checked by content: the added and removed lines of all
three surviving files are byte-identical to their originals.
The oracle, and how its counts were read
Kafka publishes its compiled tests to Central. A dedicated surefire execution points at
StreamTaskTest,RecordCollectorTestandProcessorContextImplTest, which therefore exerciseour patched classes, with nothing excluded, relaxed or recompiled. It runs in the normal
testphase on every build.
Locally, the whole module
testphase was run and the counts read out of the report XML inparallel-consumer-streams/target/surefire-reports-kafka-upstream/- not from the consolesummary, and not with
-Dtest=, which silently overrides that execution's<includes>so the suitedoes not run at all while the build still goes green. All three classes are green with zero failures
and zero skipped; the per-class numbers are deliberately not written here, because they move
with the patch, with
kafka.version, and again when the seam arrives - re-derive them.Three other checks, each with its control:
ShadowedClassLoadingTestchecks eachgenerated class loads from
/classes/and not a.jar; thatStreamTaskandStreamThreadstill come from the jar; and that both sets share a package name and a classloader.
StreamTaskis the sharp choice - it is the class the seam patches next, so the assertion has tochange visibly on the day the generated set does.
released sources.
patched classes.
Publication is off, deliberately
maven.deploy.skip,gpg.skipand the publishing plugin'sskipPublishing. Merging an isolatedleaf module is cheap to reverse; publishing an artifact carrying classes in Apache's own package
namespace is not - classpath order is a convention rather than a guarantee, class loading is per
class so the result is always a mixture of two versions, and JPMS forbids split packages outright.
maven.install.skipis not set, so a local install still works.NOTICEcarries the Apache 2.0section 4(b) statement naming the three modified classes.
Two things cutting fresh from master surfaced
Neither exists on the origin branch, whose base predates both gates:
RequireUpperBoundDepsfails onjackson-annotationsoncejackson-databindispromoted ahead of what
kafka-streamsresolves. Declared at the matching version.forbiddenapisfires on locale-dependentString.formatcalls in Kafka's released source, atsites the patch does not touch. The generated tree is excluded rather than patched - fixing them
would mean patching Apache code for a rule Apache never adopted, growing the change-set this
design exists to keep small. Anything of ours landing later sits under
bz.stuband is notcovered by that exclusion. Same scoping decision ArchUnit already makes here.
A third was found on the way in: the origin branch's
logback-test.xmlfor this module would failbin/check-test-log-config.sh(root pinned toinfo, a strayio.vertxdebug logger). The module'sconfig is written to the gate instead, and the module is registered with the gate - which exposed
that the gate's self-test held a second copy of the module list and inverted when the two
disagreed, reporting four failures for a gate that was working. The self-test now reads the list out
of the gate.
Where this came from, and what it de-risks
Copied out of #271, the Kafka Streams feasibility study, which keeps the
execution seam, the measurements and the semantic work. This is the first rung of the reconstructed
Wagon B stack: the branch forest stays as the evidence record, and each rung is cut fresh from
masterrather than merged, so the PRs document what the design is rather than how it wasdiscovered.
It also de-risks #269 (Kafka Connect on PC), which uses the same
shadowing technique against a different Apache module and piggybacks this machinery and its
write-up.
What is deliberately not in this PR
docs/data/roadmap.yaml'sstreams-parallelism-previewstaysat
limited-pocwithpull_request: astubbs/parallel-consumer#271. The stages block orders theladder by the most advanced artifact that exists, and that is still the spike; this PR carries no
per-key parallelism, so nothing here advances the track. Its
done_when- an opt-in alpha modulewith setup a reader can follow - is not met while publication is off.
each module", which read literally makes this the trigger; moving them would assert
opt_in: artifact dependencyfor an artifact that publishes nothing. Both now say publishablerather than landed, and the reasoning is written at the point where somebody would otherwise
move them.
One inherited sighting recorded, not fixed
The first CI run's Chaos lane went red on
ChaosRevokeUnderWorkCooperativeDrainIT, seed1159274055608047297- the standing revoke-pathcommit deadlock, with the same discriminator its ledger was written to distinguish (poll thread
BLOCKED acquiring a monitor held by
pc-controlinsidecommitOffsetsThatAreReady, reached fromonPartitionsRevoked). That is core's, tracked indocs/inflight/bug-857-family.mdand owned by#29; this diff touches no core Java, no locking and no rebalance path.
It did not reproduce on the second run, and that is recorded as intermittency rather than as a
clearance. The two Chaos runs here are separated only by documentation commits, so this is a
same-tree pass/fail pair - 1 of 2, the closest-together one that ledger has. Both the seed and
the pair are written down there, because the log expires and the seed is the asset; nobody has
replayed it.
The Unit and Integration lanes failed on that same first run for an unrelated reason: a Maven
Central read timeout resolving
org.apache.kafka:kafka-streams:jar:sources:3.9.2. Both are greenon the re-run, so it was transient - but it is a real new dependency of CI, since the machinery
needs a sources jar and nothing here fetched one before. Worth knowing if it recurs.
Checklist
it unpublished), the technique write-up in
docs/solutions/architecture-patterns/,NOTICE,AGENTS.md's module table, and the three records this module's arrival made inaccuratedocs/features/-N/A - there is no user-facing feature here and no artifact to depend on. The held record and its trigger are corrected in docs/inflight/release-experimental-module-records.md insteadShadowedClassLoadingTest(the premise everything rests on),TestConventionsArchTest, Kafka's own suite as the behaviour-preservation oracle, and thelog-config gate's self-test taught to read the module list from the gate
ce-simplifyandce-code-reviewlocally -N/A - spending the review on the PR instead; the diff is a module extraction whose correctness claim is the upstream suite, which runs in CI