Repository navigation
Conversation
A requirements-only plan from a brainstorm on this branch. Feasibility proof, not a shippable capability, and it does not move the admin-then-producer-then-consumer shipping order. Three findings changed the shape of it during the dialogue, and each is recorded where it bites: THE TOPOLOGY IDL WAS THE WRONG FRAMING. A topology is a fluent chain of JVM builder calls, so the wire needs handles and method names, not a description of streaming semantics. Owner's call: do not insulate the Kafka Streams API - expose it, and accept the coupling that creates. The PoC takes real handles with a fixed five-method set, so the mechanism is shown to generalise without the argument type system landing at PoC stage. THE PROTOCOL HAS NOWHERE TO PUT A RETURNED VALUE. Verified against the frozen v1 schema: Report.Success carries only records for the proxy to PRODUCE, which is not the same as handing a value to a downstream operator, and Report's other arms are outcomes. So this needs a new correlated request/response pair, and that - not the topology description - is the bulk of the work. A STREAMS OPERATOR IS SYNCHRONOUS, SO THE ENGINE BLOCKS A STREAMS THREAD ON THE FOREIGN CALL. In Parallel Consumer the core thread is never held; here it must be. Concurrency is therefore num.stream.threads, bounded by partitions, NOT by key count - so this demonstrates the capability and not the key-parallel pitch, and the plan says so in three places rather than letting a reader infer otherwise. Key-level parallelism would come from Parallel Consumer beneath Streams (#255), which is assumed not to land and which nothing here may depend on. The Problem Frame carries real demand evidence rather than the assumption I had queued: the owner was asked repeatedly, while consulting for Confluent, whether Kafka Streams would come to other languages, and had to answer no each time. Also records in STRATEGY.md a consequence of the wrap-the-client fork that inverts something that section already says. It positions Parallel Consumer AGAINST Share Groups on per-record overhead; that comparison stands, but wrapping the client means non-JVM languages would GET Share Groups from us, early, because they land in the Java client first and librdkafka follows. The segment the doc already names as "who will want Share Groups first" becomes one to serve rather than argue with. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
…heaper Everything in Other runtimes pointed one way: those ecosystems have no Kafka Streams, so give them Kafka Streams. The same mechanism points the other way, at an audience that is larger and already paying for Kafka. Kafka Streams users are on the JVM, and the JVM has no PyTorch, scikit-learn, transformers or pandas. Today they either export a model into something the JVM can run, which constrains what they can build, or they stand up a service and call it over the network from inside the topology. Two things make this worth writing down rather than filing as an idea. OUR CONSTRAINT IS ALREADY THEIR STATUS QUO. A Streams operator is synchronous, so an HTTP call from inside one blocks the stream thread exactly as a foreign-function call would, with the same rebalance exposure when it runs long. The blocking-thread limitation reads as a compromise only against JVM Streams; against what these teams actually do today it is the same blocking minus the network hop and minus the second deployment. AND IT NEEDS A STRICT SUBSET OF THE WORK. A JVM team writes its topology in Java, so there is no topology description to carry and no handle protocol to design - only the per-record function, which the language proxy already carries in four languages. The non-JVM direction needs everything; this one needs only the shared core. So the cheapest thing to build is a complete mechanism for one audience and the foundation for the other. The counterweight is recorded rather than omitted: network model serving buys independent scaling, GPU pooling, versioning and canarying. The teams to sell this to are the ones who stood up a Flask service because they had no other way to get a Python function into a stream, not the ones already invested in a serving platform. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
Five reviewers (coherence, feasibility, product-lens, scope-guardian, adversarial) returned 21 actionable findings against the Kafka Streams requirements. Applied as one pass rather than 21 patches, because five of them resolve into a single change and would have contradicted each other piecemeal. THE STRUCTURAL ONE. A Streams session cannot open on the v1 wire at all: the handshake demands a topic list or pattern and builds a Parallel Consumer engine from it, while a Streams topology names its own sources. Verified against OptionsMapper, not inferred. Combined with the wire being additive-only - so a message shape chosen to keep an argument type system OUT of a proof would become permanent contract every client library inherits - the PoC now defines its own experimental package and handshake and does not edit the frozen file. That also defers the specification- coverage build check, which fires the moment an undocumented message lands in the frozen proto and which the plan had not named at all. THE ORACLE WAS UNRELIABLE IN THE EXACT CASE THE PLAN EXPECTED TO SEE. Four reviewers converged. Exactly-once is deferred, so the engine runs at-least-once; a rebalance replays records and inflates counts; and the plan explicitly planned for a rebalance. A correct implementation could have gone red indistinguishably from duplicate delivery. The count oracle now asserts a rebalance-free window, and the slow-function example is separated and reports no counts. AND THE ORACLE COULD NOT TELL THE PLAN APART FROM DOING NOTHING. Every acceptance example passed unchanged on Streams -> topic -> Parallel Consumer -> topic -> Streams, which needs no protocol work and already exists here. That baseline is now recorded as considered-and-rejected for the one reason that survives - the topology stays Java-defined - and a new acceptance example makes the definition path itself the oracle. Corrections to claims I wrote and could not support: - "The only thing that must cross is the user's function" was false. The topology description crosses too, and the plan's own cited source calls that the bulk of the work. Fixing the method set at five bounds that problem rather than solving it, and the Problem Frame now says so. - "A capability those ecosystems do not have at all" is refutable in five minutes - Faust, Bytewax, Quix Streams, goka. Narrowed to what holds: none of them is Kafka Streams. - The demand evidence establishes category interest, not that anyone would accept a sidecar, a replayed Java builder API, or a blocking per-record hop. Nobody was asked. - "Sidecar, not embedded" claimed to avoid RocksDB under GraalVM. It does not - the sidecar already ships as a native image - so the rationale is rewritten and the JVM-sidecar assumption is recorded. A kill criterion now exists, written before building, per the sibling shared-C-transport plan's precedent. The inverse audience is recorded as an actor with the subset of requirements that serves it. The dependency's Jackson CVEs are named along with why the obvious pin is blocked. Requirements were renumbered to stay continuous after the additions, and every Governs and Covers link was re-pointed with it. One decision was deliberately not made: whether to sequence the PoC in two provable halves. That is a scope call and it is in Outstanding Questions rather than answered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
… two research passes Two research passes turned three of the requirements' open questions into decided ones, each on evidence rather than judgment. WHERE THE SECOND SERVICE LIVES. ProxyServer accepts one bindable service and constructs one connection guard per start, and it deliberately compiles against no generated protocol type. Hosting two services there would force a choice about whether the single-connection slot is shared across them, and would breach that one-way seam. A second entry point is the established pattern here - the test-mode sidecar already is one - and gives the Streams session its own server, guard and lifecycle without touching the transport. WHERE THE EXPERIMENTAL SCHEMA LIVES. Three gates decide it. The specification-coverage test reads only the descriptor that declares Configure, so a second file is invisible to it - which mechanically confirms R7's claim. The lint gate runs over the protocol module's whole proto root. The breaking-change gate refuses to run at all if it finds a second copy of the frozen path. The new module owning its schema keeps all three clean. AND ROCKSDB IS OUT, WHICH IS A CHANGE OF POSTURE RATHER THAN A CONVENIENCE. Verified externally: Kafka Streams under native image is solved territory, but its documented failures are per-platform and land on this project's own - a container build hardcodes a linux64 native library, the artifact ships Linux binaries only, and an older release was not compiled for macOS arm64. An in-memory store exercises state, changelog and commits while removing that component entirely, so the native-image question becomes INDEPENDENT of this PoC rather than deferred by it. Four traps from the off-branch Kafka Streams corpus are now designed around rather than discovered: core's tests jar leaks a parallel-execution property onto any consumer's classpath and failed 159 of Kafka's own tests on state-directory lock contention; the root build sweeps target/generated-sources as a test source root so generated classes get compiled twice with the test copy shadowing; -Dtest= silently overrides an execution's includes so a scoped run reports green having run nothing; and the whole patch-and-shadow mechanism the existing Streams module uses is inherited by nobody who sticks to the public builder API. The plan also records that the frozen wire's additive-only gate baselines against a moving reference, so it catches drift within a change rather than protecting the contract over time. Nothing here depends on it, but the requirements implied more than the gate delivers. Eight units, each with named files, test scenarios and a verification signal. The invocation registry is marked test-first: it is the one piece whose concurrency defect is invisible to a single-threaded run. One judgment call, stated rather than buried: the second ce-doc-review pass was skipped. The requirements half came out of a five-persona review under an hour ago with all findings applied; the implementation units are consequently unreviewed, and that is the accepted cost. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
…(U1) The module a non-JVM program's topology will run inside. Nothing behavioural yet - this is the container, its dependency posture, and the traps designed around rather than discovered. Kafka Streams needs Java 11 or newer, so release.target is overridden to 17. That override is deliberate rather than incidental: the release level constrains the platform API and not what javac reads off the classpath, so a module left at the project default compiles happily and ships bytecode claiming a JVM it does not run on. The proxy and mutiny modules record the same trap. STOCK KAFKA STREAMS, PUBLIC BUILDER API ONLY. The existing patched streams module reaches Kafka internals by unpacking and patching it at build time, which carries the parked two-jars-on-one- classpath problem: classpath order is a convention rather than a guarantee, class loading is per class so the result is always a mixture, and split packages are illegal on the module path. A module that ships no class in an org.apache.kafka package inherits none of it. This one does not unpack, patch or shadow anything. The Jackson pin is module-local and the reasoning is copied from the streams example, because the constraint is the same: databind is also transitive to WireMock, and hoisting the pin into root dependency management forces WireMock onto an incompatible Jackson and breaks the Vert.x tests. The BOM rather than a bare databind pin, because databind needs core and annotations on the same line. This module publishes nothing, so nothing downstream inherits it - which is also what keeps Kafka Streams out of the artifact the admin wrapper ships from. Test parallelism is pinned off in surefire. Core's tests jar places a parallel-execution property at the classpath root of every module consuming it, and Kafka Streams tests are written for a serial runner - it failed 159 of Kafka's own tests on state-directory lock contention. Surefire configuration parameters outrank the property file. Both docs-data fragments land with the module rather than after it. The maturity fragment alone was not enough: its evidence id has to resolve in the testing-evidence corpus, and bin/check-docs-data.sh - which is not part of the Maven build - said so. Verified: 3 tests run, 0 failures, read from the surefire report rather than the build's exit code. Copyright gate run after staging, since it only sees tracked files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
The schema a foreign runtime speaks to describe a topology, and it deliberately does not live
beside the frozen v1 wire.
WHY IT IS SEPARATE, AND WHY THAT IS FORCED RATHER THAN TIDY. The v1 handshake rejects a Configure
naming neither a topic list nor a topic pattern, and builds a Parallel Consumer engine from that
subscription - so a Streams session, whose topology names its own sources, cannot open on it
without fabricating a subscription and standing up an engine nothing uses. Separately, v1 is
additive-only, and a message shape chosen to keep an argument type system OUT of a proof has no
business becoming permanent contract every client library inherits.
Keeping the schema in this module also keeps three gates clean, and each was checked rather than
assumed: the specification-coverage test reads only the descriptor declaring v1's Configure, so
this file is invisible to it; buf lint runs over the protocol module's proto root, which this is
not under; and the breaking-change gate refuses to run at all if it finds a second copy of the
frozen path - there is still exactly one.
The message set is the whole thing the flows imply, not one pair: an open handshake, a builder
call per method with a client-chosen call id echoed on the handle it is answered with, function
registration, describe-complete, and the correlated invocation pair. Handles are server-minted and
opaque. A result carries either a value or an error, never both - a failed foreign function fails
the record rather than feeding a wrong value into an aggregation.
Codegen writes outside target/generated-sources for the reason the protocol module records: the
root build sweeps that path as a test source root, so anything generated there is compiled twice
and the test-classes copy wins on the surefire classpath. GeneratedCodePlacementTest asserts the
class loaded is not the shadowed one, rather than asserting a file is absent from a directory -
the latter passes while a duplicate elsewhere shadows it.
Verified: 12 tests, 0 failures, counts read from the surefire reports. buf lint passes - it was
exiting 2 ("has NOT run") until buf was installed, and an unrun gate is not a passing one.
The breaking gate still cannot run here, and that is recorded as its own defect rather than shrugged
at: it uses a bash 4 builtin and macOS ships bash 3.2, so it exits 127 before checking anything. CI
is unaffected. The substantive assurance stands on its own - the frozen file is unmodified and has
exactly one tracked copy, both checked directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
The piece that hands a record to the host and waits. Built test-first because the plan says so, and the plan says so because this is the one place where a concurrency defect is invisible to a single-threaded run: a result delivered to the wrong waiting thread produces a plausible wrong answer, not an error. Red observed before implementation: 5 tests, 1 failure and 4 errors, every one an UnsupportedOperationException from the stub. Green after: 17 tests across the module, 0 failures, counts read from the surefire reports. THE TEST THAT EARNS ITS KEEP answers each concurrent invocation with a value derived from the payload that invocation sent, then asserts every caller received its own echo. A count-based assertion - eight callers, eight results - passes just as happily when the registry crosses two waiters, which is the defect worth catching. Three details that are deliberate rather than incidental: - The waiter is registered BEFORE the invocation is emitted. A host fast enough to answer during the emit would otherwise find no waiter and have its result discarded as an unknown correlation. - The pending entry is removed in a finally, including on the timeout path. A registry that leaks on timeout grows without bound under exactly the conditions that caused the timeout. - An unknown correlation is discarded rather than applied to anything else, and that is reachable without a bug: a result arriving after its invocation timed out has nobody left to answer. An error from the host fails the record rather than substituting a value. A failed record is visible; a substitute value entering an aggregation is a wrong count nobody can trace. The mapper is a ValueMapperWithKey rather than a ValueMapper so the key travels too - the invocation message carries one, and a host that cannot see the key cannot make a keyed decision. It blocks the stream thread for the whole round trip, which is a property of Kafka Streams operators being synchronous rather than a choice made here, and is why concurrency is bounded by stream-thread count rather than by key count. The transport sits behind a one-method sink so the registry and the mapper are testable without a gRPC stream anywhere near them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
The engine holds the builder objects and hands back opaque handles; the host names them back. Nothing here hardcodes the chain's shape, which is the claim the whole proof rests on - the topology produced is whatever the host described. The end-to-end test is the definition-path oracle in unit form: five calls issued as a host would issue them, then the built topology actually counts per key through a test driver. Errors name the offending thing - the handle, the method, the kind it needed - because a foreign caller never sees a stack trace, only the text that crosses the wire, and "invalid request" cannot be debugged from the other side. A RED-PROOF CAUGHT A TEST THAT PROVED NOTHING, which is the part of this commit worth reading. The store-choice assertion is the one guarding the plan's headline decision, so it was sabotaged to check it could fail: swapping the in-memory store for a persistent one and re-running. IT STILL PASSED. TopologyTestDriver deletes the state directory on close, so a walk after the try-with-resources reports an empty tree for a persistent store too - the check could not fail under any circumstance. Fixed by inspecting while the driver is open. Re-sabotaged: now it fails. Restored: green. The assertion discriminates, and it did not before. Two things fell out of that. RocksDB does load and write on macOS arm64 with this Kafka version, so the platform problems recorded against native image are about native image and not about the JVM path. And the sink of a count is a changelog carrying every intermediate value per key - the test reads last-value-per-key, and the demo will have to read it the same way or report wrong counts on a healthy run. Verified: 23 tests across the module, 0 failures, counts read from the surefire reports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
…int (U5) Ties the assembler and the invocation bridge to a gRPC stream, and gives the result its own entry point beside the Parallel Consumer sidecar. A SECOND ENTRY POINT RATHER THAN A SECOND SERVICE ON THE EXISTING LISTENER. The proxy's server takes one bindable service and constructs one connection guard per start, and it deliberately compiles against no generated protocol type. Hosting two services there would force a choice about whether the single-connection slot is shared across them, and would breach that seam. The test-mode sidecar is this repo's own precedent for adding a differently-configured entry point instead. This one gets its own server, its own guard and its own lifecycle for free. From a client's point of view it behaves like the existing sidecar: the port is the first line on stdout, using the proxy's own prefix constant so the two cannot drift, and the parent-death watchdog is reused rather than reimplemented. The watchdog matters more here than there - a leaked Streams instance holds state-store locks as well as group membership. The dependency arrow is streams -> proxy, which is the safe direction. The constraint this module lives under forbids proxy -> streams, so that Kafka Streams cannot reach the artifact the admin wrapper ships from. Depending the other way costs nothing downstream, because this module publishes nothing. Inbound needs no locking - gRPC serialises a stream's callbacks - but outbound does, and that asymmetry is the whole reason the send is behind a lock: several stream threads emit invocations while the transport thread may be answering a builder call. Every refusal names what was refused: which message arrived before Open, which call id named no method, which handle does not exist. A foreign caller never sees a stack trace, so a refusal that says only "invalid request" cannot be debugged from the other side. A refusal faults the message rather than killing the stream, which the tests assert. Javadoc is skipped for this module, and the reason is mechanical rather than laziness: the hand-written classes import protocol types generated into this module's own target, which is on the compile path but not on javadoc's source path, so the delombok step cannot resolve them. That is safe only because nothing is published here; a publishing module would have to fix the source path instead. Verified: 29 tests across the module, 0 failures, counts read from the surefire reports. The session state machine is driven with a recording observer, so everything up to describe-complete is covered without a broker or a server; a runner that throws makes an accidental topology start loud rather than silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
Gives Python a topology-building surface and a function registry, so a Python process can describe a Kafka Streams topology and then serve the transforms inside it. The topology is built by replaying builder calls over the wire; the transforms stay in Python and are reached by token. What crosses the boundary is a token, never a function pointer. `register()` files the callable in a local dict and sends only its token; when the engine reaches that node it sends an `Invocation` carrying the token, and the reader thread dispatches it back to the callable. Neither side ever needs the other's address space, which is what makes the same shape work over a socket today and over the C ABI later. `TopologyBuilder` calls block until the engine answers with the handle it assigned. That is deliberate: handles are the engine's to mint, and a client that guessed them would silently diverge the moment the engine's numbering changed. Blocking keeps the two handle tables identical by construction. A transform that raises reports the error back over the correlation rather than substituting a value. Silently answering with a default would turn a broken transform into plausible-looking output, which is the worst failure mode available to a stream processor. The same holds for an invocation naming a token nobody registered: it is answered with an error, not dropped, because dropping it would hang the engine's mapper on a reply that never comes. The generator now covers both schemas. They stay separate modules rather than merging into one: `proxy.proto` is the frozen wire every client shares, and `streams.proto` is experimental and may change or disappear. Each generated stub's header now names the proto it came from, so a reader can tell which of the two they have open without going back to the generator to find out. Tested against a fake engine rather than a live sidecar, so the tests exercise the protocol's ordering rules - the reply to a builder call, an invocation arriving unsolicited, a fault mid-build - without needing a broker. The live path is U7's job. Verified: 6 new tests pass, ruff clean, mypy --strict clean over 19 source files, generated stubs match their schemas, copyright gate 0 violations across 906 files against the fork point.
One command builds the engine, starts a broker, seeds a backlog, describes a
topology from Python and checks the counts that come out:
demo/run.sh --streams --native
It works. 1000 keys counted exactly, by a topology whose every operator was
named from Python and whose transform ran in Python on every record.
The counts are checked rather than displayed. The demo seeds the backlog, so it
knows the answer, and it reads the sink LAST-VALUE-PER-KEY: a count is a KTable
changelog, so a key whose final count is 12 carries 1 through 11 ahead of it.
Summing the topic would report 78 for that key and call a correct run broken.
Two things this unit found by running, which no amount of unit testing would
have surfaced:
A clean shutdown printed a CANCELLED fault on every run. Closing the transport
breaks the response stream, and the reader could not tell that breakage from an
engine failure - so it reported the close as an error. A session now knows it is
closing before it tears the transport down. Fixed with tests on both sides:
a close must be silent, and a break we did NOT ask for must still fault. Both
tests were confirmed to fail against a sabotaged fix, in each direction.
The rebalance check was worthless as first written. It sampled the consumer
group once at the end, and a rebalance is a transient - a single-member group
that gets evicted rejoins within a second and reads STABLE again. It would have
pronounced a run rebalance-free and said exactly the same thing about a run that
rebalanced twice in the middle. It now samples throughout, on its own thread,
and excludes the samples taken while the engine is still joining, because that
is the engine arriving rather than a rebalance.
The failure arm refuted its own prediction, and says so. It was written
expecting a transform slower than max.poll.interval.ms to get the engine evicted
from its group. That is not what happens: Kafka Streams interleaves its polling
and keeps its membership, and the symptom is throughput collapsing to about
three records a second and the run not finishing. The arm reports what it
observed rather than what it expected. It still refuses to accept a WRONG count
in either arm - missing means the run did not finish, wrong would mean something
counted incorrectly.
The timing figures measure the invocation window - first entry to last exit -
not the whole run. Including engine startup and the verifier's polling made the
first draft report a per-invocation cost four times worse than the boundary
actually is. As measured here the round trip is a few hundred microseconds and
the Python function is a rounding error inside it, which is the useful finding:
the boundary is the cost, not the language.
The JVM resolver moved to demo_jvm.py rather than being copied. Two demos that
disagree about which JVM they ran on would quietly invalidate any comparison
between them, and the second copy would have drifted the first time one of them
learned a new way to find a JDK.
--streams --docker is refused rather than ignored: the demo image's entrypoint
is the comparison demo, so a container run would build, start and report the
wrong demo while looking like it had honoured the flag.
Verified: both arms run green through run.sh; the comparison demo still runs
unchanged after the resolver refactor; 8 tests pass; ruff clean across the whole
module for the first time (a pre-existing overflow in reference_demo.py's prose
is rewrapped here because this commit is what makes that file's lint state mine
to leave green); mypy --strict clean over 20 source files and --check-untyped-
defs clean over 13; copyright gate 0 violations across 909 files against the
fork point.
…ted (U8) The note was written as "a proof of concept to run". It has been run, so it now records what running it taught - including where the note's own predictions were wrong, which is the part worth keeping. Handles beat an IDL, and that is the headline. The note's central claim was that a topology IDL "would have to be designed" and would be the bulk of the work. It did not have to be. Each builder method is one message, the engine performs it against a live StreamsBuilder and returns a handle, and the next call names that handle - so the topology is never described, it is built remotely one call at a time. The wire carries no model of what a topology is, which means there is no shared model to keep in step as Kafka Streams adds operators. The kill criterion's first condition is met: a sixth method taking scalar arguments is one more member of a oneof. The condition that is NOT met is an argument that is itself behaviour - a serde, a comparator, a joiner - which needs the function-token pattern generalised. A slow foreign function does not break the consumer group. This was a confident prediction and the experiment refuted it: a 300ms transform against a 5s poll interval left the group STABLE with one member for all 78 samples, because Kafka Streams interleaves its polling. It fails as collapsed throughput instead. The relief is that slowness is not a liveness hazard; the new question is that it fails as silence, and nothing on the wire distinguishes "still working" from "stuck". Serdes are a non-issue except where an operator mints a value the host never supplied. count() produces a Long, so the sink must write Serdes.Long() and the host must decode eight big-endian bytes. The engine special-cases it, and that does not generalise - every aggregation, reduce and join has the same shape. Typed handles are named as the next real design question, ahead of any of the deferred operators. The protocol cannot report engine state. The demo wanted to claim its run was rebalance-free and had to ask the Kafka admin API from outside, which is a strange thing to need when the premise is that the host does no Kafka I/O. Recorded with the trap that cost time here: a rebalance is a transient, so a sample taken at the end pronounces a run clean and says the same about a run that rebalanced twice in the middle. The measured cost is 400us per round trip with the Python function at 0.2us of it - 0.05%. Written down in both directions: optimising the foreign function is pointless and only the crossing is worth attacking, and the ceiling is per stream thread, which JVM Kafka Streams is bounded by too, so it is parity rather than a deficit. Every deferred capability now carries what it would actually need, in a table, rather than being a list of things left out. Punctuators are confirmed cheap. Exactly-once is named as a genuine open question rather than a plumbing task, because what a foreign function's side effects mean inside a transaction is not something the protocol can answer. The pre-PoC guesses are kept rather than rewritten, marked as written-before, because the ones that turned out wrong are worth being able to see. Verified: docs-data check 86 files structurally valid; copyright gate 0 violations across 909 files against the fork point.
Dependency ReviewThe following issues were found:
License Issuesparallel-consumer-proxy-streams/pom.xml
OpenSSF ScorecardScorecard details
Scanned Files
|
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
|
| PR | Base | Change | |
|---|---|---|---|
| Clones | 169 | 155 | :face_with_monocle: +14 |
| Duplicated lines | 2843 | 2708 | :face_with_raised_eyebrow: +135 |
| Duplication | 1.24% | 1.24% | ➖ 0 |
| Rule | Limit | Status |
|---|---|---|
| Max duplication | 2% | ✅ Pass (1.24%) |
| Max increase vs base | +0.1% | ✅ Pass (+0.00%) |
⚠️ 16 new clones introduced
- 11 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:167<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:132 - 8 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:204<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:186 - 12 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:264<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:232 - 11 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:447<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:413 - 12 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:464<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/WindowedAggregatorCallCountTest.java:425 - 7 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/StreamsSessionServiceTest.java:162<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/StreamsSessionServiceTest.java:105 - 7 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/StreamsSessionServiceTest.java:226<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/StreamsSessionServiceTest.java:107 - 7 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/StreamsProtocolRoundTripTest.java:18<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/StreamsSessionServiceTest.java:12 - 11 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/InvocationRegistryTest.java:123<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/InvocationRegistryTest.java:91 - 9 lines:
parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/InteractiveQueryTest.java:222<->parallel-consumer-proxy-streams/src/test/java/bz/stub/parallelconsumer/streams/InteractiveQueryTest.java:78 - 9 lines:
parallel-consumer-proxy-streams/src/main/java/bz/stub/parallelconsumer/streams/ForeignReducer.java:30<->parallel-consumer-proxy-streams/src/main/java/bz/stub/parallelconsumer/streams/ForeignValueMapper.java:28 - 9 lines:
parallel-consumer-proxy-streams/src/main/java/bz/stub/parallelconsumer/streams/ForeignJoiner.java:28<->parallel-consumer-proxy-streams/src/main/java/bz/stub/parallelconsumer/streams/ForeignValueMapper.java:28 - 9 lines:
parallel-consumer-proxy-streams/src/main/java/bz/stub/parallelconsumer/streams/ForeignAggregator.java:27<->parallel-consumer-proxy-streams/src/main/java/bz/stub/parallelconsumer/streams/ForeignValueMapper.java:28 - 9 lines:
docs/inflight/next-batching-modes-for-clients.md:119<->docs/inflight/perf-crossing-fixed-versus-per-byte.md:55 - 20 lines:
parallel-consumer-proxy-protocol/pom.xml:80<->parallel-consumer-proxy-streams/pom.xml:164 - 22 lines:
parallel-consumer-proxy-protocol/pom.xml:158<->parallel-consumer-proxy-streams/pom.xml:240
Powered by astubbs/duplicate-code-cross-check
|
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
astubbs
marked this pull request as draft
August 22, 2026 22:53
Adds a Describe request and a TopologyDescription answer, so a host can ask what the topology it described actually looks like. The host does not already know. It issues builder calls and holds opaque handles; the engine is the only side that has ever seen the assembled graph - the nodes Kafka Streams generated itself, the sub-topology split it chose, and whether an aggregation needed a repartition. Running the demo now prints a KTABLE-TOSTREAM node that nothing in Python asked for. The answer carries the same graph twice, on purpose. The text field is exactly what Topology.describe() produces, and that is a compatibility surface rather than a convenience: every Kafka Streams topology visualiser in existence parses that text, so a host able to print it can use all of them. The structured form is there so this project's own tooling never has to parse that text back - it is a human-readable rendering with no stability guarantee, and each of those visualisers has had to reverse-engineer its shape. This is the argument for wrapping the real engine rather than reimplementing it, reduced to one artifact. None of the Python-native stream processors has a describe(), so none of them has any of this tooling, and none can grow it without also growing an ecosystem. We inherited it for the cost of one message. Building the topology is now memoised rather than one-shot. Describing has to materialise the topology, and under the old one-shot build a host that asked what it had built was then refused permission to start it - punished for looking. Node ordering is sorted because Kafka Streams returns nodes in a Set, so without it a description reorders itself between runs and any diff of two descriptions is unreadable. A prediction this refuted, in the tests: grouping before an aggregation was expected to force a repartition and split the topology in two. It does not. mapValues cannot change the key, so Kafka Streams knows the existing partitioning still holds and skips the repartition entirely. The test now asserts the true fact and doubles as a regression guard - widening mapValues into a general map, which CAN change the key, would silently add a repartition topic and a broker round trip per record, and that test fires. Two Python tests were worthless when first written and are rewritten. The fake engine answered inside send(), before describe() began waiting, which hides exactly the two defects the tests exist for: an answer already sitting there satisfies a stale event as well as a fresh one, and a waiter that never waits cannot be woken. Both passed against deliberately broken code. They now use a fake that answers late and differently each time, and both were confirmed to fail against their sabotage - the missed-wakeup one by hanging to its timeout, which is the symptom it exists to catch. Faults now release a describe waiter, not just the handshake. A waiter left blocked on a session that has already failed turns a reported error into a timeout, and at the call site that reads as a hang - which gets diagnosed as a network problem rather than as the error the engine did send. Verified: Java suite green including eight new describer tests, the memoisation red-proofed; Python suite green with the two rewritten tests red-proofed in both directions; ruff and mypy clean; demo runs end to end and prints the topology; copyright gate 0 violations against the fork point.
…rs that got there first Records the findings from the Kafka Streams proof of concept in the places that decide what gets built, and corrects three claims the run falsified. STRATEGY.md said the hard part was that a topology has no portable description and that an IDL would have to be designed. The PoC did not design one and did not need one - replaying builder calls against a live builder leaves no shared model to drift. The caveat is recorded beside the claim rather than buried: cheapness is bought by giving up inspectability, and Beam chose an IDL precisely because a runner must reason about a pipeline to optimise it and to run it on more than one engine. If cross-engine portability ever matters, the IDL returns. Three positioning findings that were not previously written down anywhere. Nothing of the user's code crosses our boundary, and that beats the obvious comparison. Beam pickles the DoFn into the pipeline proto because its harness may run on another machine, which is where Python's constraints on unpicklable closures and resources come from. Our harness is the host process, so we send a token and call the function in place - a user can capture a database handle, an open socket, or a loaded model. For the machine-learning audience the strategy already targets, a loaded model is exactly the thing that cannot be pickled into a pipeline, so this is a headline rather than a detail. The nearest shipped competitor is PyFlink, not a Kafka project. PyFlink and PySpark are the same shape as this work - a Python API over a JVM engine, Py4J for the control plane - and both need a cluster a platform team operates. Kafka Streams is a library that runs in the user's own process, so this offers embedded stateful exactly-once processing with no infrastructure, which PyFlink structurally cannot. The Python-native reimplementations are embedded but must rebuild state, windowing and exactly-once themselves. Nothing sits in the intersection. The durability argument is the one to lead with. Faust was abandoned and survives as a community fork; Bytewax became community-maintained when its backing company wound down. Reimplementing this is more than a small team can sustain, so wrapping the real engine means inheriting Apache Kafka's maintenance rather than competing with it - and inheriting its tooling, which U9's Describe makes concrete. Settles the recurring "why would Java use this" question, and the answer is not the expected one. A Java team has all of it natively and a boundary crossing is pure loss, so virtual threads argue for using PC directly rather than through a proxy. What does hold is version decoupling: the sidecar runs its own JVM and its own client, so an application pinned to an old JDK or an old client jar reaches modern broker features without upgrading itself. Share groups are the worked example and the reasoning is easy to get backwards - they need broker support, but they need a new CLIENT too, and that is the half a legacy application cannot supply. Scoped honestly as a legacy-estate argument, not a reason for greenfield Java to adopt a proxy. The landscape note gains PyFlink and PySpark as first-class entries, since they are the same shape as this work rather than merely adjacent. Both split control plane from data plane: Py4J for control in both, which is a handle model like ours but reflective and untyped, and therefore permanently tied to a JVM at the far end where ours can become a C ABI. On the data plane both have already moved off per-record IPC - PySpark to Arrow-batched vectorized UDFs, PyFlink via FLIP-206 to a thread mode where PEMJA embeds the interpreter in the JVM process. Both escape routes this project is weighing have been taken by someone larger, and neither project picked one and stopped, which suggests the answer is "both, selectable" rather than a choice. Also corrects the note's Beam entry: Beam does not pass function references, it passes the function. Describing the two mechanisms as the same thing was wrong. Records why embedding an existing Kafka Streams visualiser is blocked, since the obvious next thought after Describe is to put one in a web UI. Checked on the repositories directly: two of the tools declare no licence at all, which means all rights reserved; KSTD is GPL-3.0, which would force this Apache-2.0 project to GPL-3. Linking users to them is free; vendoring them is not. If a diagram is ever wanted in our own UI the renderer should be written against the structured form Describe already returns, which skips the ASCII parsing every one of those tools had to reverse-engineer. Verified: copyright gate 0 violations against the fork point, issue-reference gate clean including the PR body, docs-data structurally valid.
…nding makes Adds docs/language-bindings.md, an inflight for rendering the topology, and the single-import surface to the strategy. The binding knowledge was spread across a plan and three inflights, and nothing said how the pieces relate. Someone designing a change that touches more than one binding had no way to find out which questions were already settled - so the Py4J and Arrow findings from the last research pass landed in a comparison document that is explicitly about other people's systems, where nobody implementing a binding would look. The new document names five axes rather than one boundary: control plane, function delivery, data plane, transport, and user-facing API shape. They are far more independent than they look - PyFlink answers the data plane two ways at once, deliberately, and keeps both. It owns nothing, and says so at the top. Every decision cites the document that owns it and that document wins on disagreement. A second copy of a rationale is a rationale that will drift, and this repo has already paid for that once. The part with no previous home is a collision. Batching the data plane is worth orders of magnitude rather than percentage points, and it is what both PySpark and Beam did. But Beam paid for it explicitly - any element failing discards and retries the whole bundle, and per-item outcomes were considered and rejected - while next-batching-modes-for-clients.md requires the opposite as a hard constraint, and per-record outcomes are a stated differentiator against Share Groups. So the obvious throughput fix would dissolve one of the product's advantages if taken the way its neighbours took it. Three options are set out; the first is that Beam's coupling of batching to bundle-level failure was a simplification rather than a necessity, and it should be costed before the others. Also records what the measured number does and does not say. A round trip of a few hundred microseconds with the Python function at about 0.05% of it means optimising the foreign function is pointless and only the crossing is worth attacking. It does NOT say whether to attack the crossing's cost or its frequency, because it was never decomposed into transport versus engine overhead. That attribution is named as the cheapest useful experiment available, and the note warns against citing the total as a transport cost. Topology rendering gets its own inflight, and its first conclusion is not to build a web UI. There is already an embedded dashboard in flight and a parked note about the sidecar hosting it; a topology view is a panel in that, and that note's constraints govern. The licensing table moves here from the Streams note, since this is where rendering now lives: two of the four visualisers declare no licence at all, which reads as permissive and is the opposite, and one is GPL-3.0. Linking users to them is free, vendoring is not. If a renderer is ever written it should target the structured form the protocol already returns, which skips the ASCII parsing all four of them had to reverse-engineer. The strategy gains the single-import surface, which is a stronger claim than any capability inside it. Each piece competes with something established; the bundle competes with nothing, because two of the five do not exist for that user at any price. Recorded with the counterweight - a single dependency carrying a JVM sidecar is a heavy install for someone who wanted only a producer, so the pieces should stay separately usable underneath the unified surface - and with the note that this argument does not apply to Java and should not be made there. Verified: copyright gate 0 violations against the fork point, issue-reference gate clean including the PR body, docs-data structurally valid.
…on in two Two corrections from review. The new note loses its "next-" prefix. Status belongs in the tags the note already carries, not in its filename. Worth recording why this keeps recurring rather than just fixing the instance: docs/inflight/AGENTS.md in this tree still teaches the opposite, with a prefix table that includes "next-", and forty notes follow it. Any agent reading the harness will keep producing prefixed filenames until that table changes, so the file rename here is the symptom and the convention doc is the cause. The owner has a commit taking status out of note filenames, but it is not reachable from this repository by sha, by message, or on any remote after a full fetch - so it appears not to have been pushed anywhere this clone can see, and the convention doc here is still the old one. Left alone deliberately: guessing at the replacement scheme risks conflicting with that commit and differing from what it actually decided. Only the status-like prefix on this branch's own new file is dropped. Then the batching question, which was recorded as one thing and is two. The language-bindings map already separated how many records cross per hop from how many the user's function is handed per call. It missed a sharper split: the two products crossing this boundary have different contracts, so "add batching" means different work in each, and neither should inherit the other's answer. For the Parallel Consumer clients, core's API is already batch-shaped - poll hands the user a context of records and a batch size of one is the degenerate case - so batching is an API-shape correction that widens the existing surface. That decision is written down and owned. For the Kafka Streams wrapper it is not the same work at all. Kafka Streams is record-at-a-time by contract: an operator takes one record and returns one value. Batching there means buffering at the operator, which changes what the operator means, and is not obviously legal for a stateful one. A stateless mapValues can buffer; an aggregation cannot without being redefined. That side has no owner, and the note now says so rather than implying the PC decision covers it. Whoever picks up batching has to say which of the two they mean in the first sentence, which is now written into the document.
…s too old to have Corrects the previous commit, which guessed. The convention it guessed at is real, documented, and already on master - this branch simply predates it. Why the earlier search came up empty, because the reason generalises. PR #323 was SQUASH-merged, so its individual commits never landed anywhere: only the squash commit a80f2bb exists on master, and c971483 lived solely on the now-gone branch. Searching for a branch commit by sha after a squash merge finds nothing and looks exactly like the change never happening. The squash commit was here the whole time, and checking the PR rather than the sha would have found it in one step. The other half is that this branch's merge-base with master is 2026-08-18 and #323 merged on the 20th, leaving it 42 commits behind. So docs/inflight/AGENTS.md as read here genuinely does still teach the prefix table - it is correct for this base and superseded on master. The claim in the last commit that the convention doc "is the cause" was right about this tree and wrong about the repository. Applying the actual rule rather than the inferred one. Filenames name an AREA and never carry state, because a name is an identifier and state changes while an identifier must not: rename on a state change and every citation breaks, leave it and the name lies. So this note becomes web-topology-rendering.md, taking the web- area prefix that master defines for the web GUI and demo, and the redundant -and-web-ui tail goes with it. State moves into the tag where it belongs. Master defines three fields with only inflight-type required, and inflight-labels is not among them, so it is dropped. The note gains inflight-state saying what it waits on, which is the fact the old next- prefix was failing to convey. Not done here, deliberately: this branch is still 42 commits behind master, and the other inflight notes it inherits still carry old-style names. Catching up is a merge from master, and it belongs on the parent branch this stack sits on rather than on a research branch at the top of it - doing it here would drag master's history into a PR based on that parent.
… and nearly all of it The PoC's 400-450us per-invocation figure was a total nobody had decomposed into the boundary crossing versus Kafka's and Streams' own per-record work, and that attribution decides whether the shared C transport is worth building. This runs the controlled experiment: a --no-transform arm builds the identical topology minus the mapValues node, so nothing ever crosses the boundary, and both arms report a sink window on the broker's log-append clock (the sink topic is now created with message.timestamp.type=LogAppendTime), which measures the engine's own production timeline in both arms alike. The finding, from 17 runs per arm sweeping 500 to 64000 records on one machine: per-record cost is only meaningful as a slope across run lengths, because both arms carry a fixed ~0.7s warm-up and cadence component that dominates short runs - at the demo's default 2000 records, an arm with no crossing at all reads ~250us per record. On the slope over N >= 8000, the treatment arm costs ~148us per record and the control arm is statistically zero (at most ~15us), so the crossing is essentially all of the marginal cost. The prediction that the crossing dominates held; the published total was refuted as a crossing price - it is roughly one third fixed ramp, and the steady-state single-thread ceiling is nearer 7,000 invocations/sec than the PoC's 2,400. Consequence for the architecture choice: attacking the crossing remains right, by C transport or batching or both, but both should be costed against ~150us per record rather than ~450us. The full method, raw windows, spread and the refuted shakedown instrument are in docs/inflight/perf-streams-crossing-attribution.md; the "What has actually been measured" section of docs/language-bindings.md now carries the answer instead of calling the attribution unfinished. The demo's default behaviour is unchanged; --no-transform is opt-in and documented in the demo README, and ensure_topic gained an optional per-topic config argument for the timestamp type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcdtMdmdhyYug3krZRWkZT
…streams-foreign-wrappers
…rom bundling Feeds the attribution measurement back into the three places that will act on it, and fixes a naming collision that was making two decisions look like one. The kill criterion had no arithmetic behind "not meaningfully faster". It has one now, and it is smaller than the plan was written against: the prize is about 150us per record, not the 450us the PoC published, because roughly 250us of that was fixed warm-up an arm with no crossing at all also pays. An embedded hop landing at 20us therefore saves about 130us, taking a per-thread ceiling from roughly 6,700/sec to roughly 50,000/sec. The bar is stated before building: if the embedded transport cannot cut the crossing to about a third of the sidecar's, the speed claim should stop being made and only the no-second-process argument survives. The same measurement is good news for that track rather than bad, and the note says so. The scenario that would have killed it - a large engine-side per-record cost hiding behind the crossing, making any transport improvement pointless - was ruled out. Kafka's and Kafka Streams' own marginal work measured at statistically zero. The measurement rule gains the lesson that nearly produced a wrong answer. Single-record-count measurement is banned here: the crossing path warms up over tens of thousands of invocations, both arms carry a fixed sub-second component, and a measurement at the demo's default scale would have reported the boundary as a fifth of the cost rather than nearly all of it. Sweep the count, take the slope, discard the warm region, report the spread. Then the scale question, because 150us is easy to wave away and should not be. At 10,000 records/sec the crossing alone consumes about 1.5 cores continuously, which is 36 core-hours a day. At 100,000/sec it is 15 cores and 360 core-hours a day. It also sets a partition floor, since Kafka Streams cannot run more stream threads than partitions - 100,000/sec needs fifteen partitions purely to have somewhere to put the boundary work, before any is allocated to the actual business logic. The cost is per record, so it is invisible below about 1,000/sec and it is the architecture above 100,000. Both directions of the comparison are kept, because either alone misleads. Against native JVM Kafka Streams this is enormous - the operator call it replaces is a Java lambda costing nanoseconds. Against what the target user actually has it is not: a non-JVM team has no Kafka Streams at all, and the JVM team's status quo for reaching a Python library is an HTTP call from inside the topology, blocking the same stream thread for a typical local round trip several times longer than this. Caveats recorded with the numbers rather than below them: the engine arm ran an in-memory store and a trivial transform, so a real RocksDB or join workload pays more per record on the engine side and the crossing's share falls even though its absolute cost does not. Records were small, so 150us is a floor for payload size. And the honest limit on what bundling recovers. The obvious arithmetic - a hundred per hop turns 150us into 1.5us - assumes the crossing is all fixed cost, which is unmeasured. A crossing is a serialise, a syscall, a thread handoff and a wake, plus a per-byte copy that bundling does not amortise. The next experiment is named: sweep payload size at fixed record count to split fixed from per-byte. Until then the note claims one order of magnitude rather than two. The naming fix. "Batching" was doing two jobs and the two were being treated as one decision. Batching now means only the Parallel Consumer client API shape - how many records the user's function is handed per call. BUNDLING means how many records cross the language boundary per hop. They are independent: a client can bundle a hundred per hop and still hand the user one at a time. Bundle is Apache Beam's word for exactly this unit, so it arrives with provenance and cannot collide. Also strips the em-dashes from the two documents added earlier in this session, which should not have had them.
…pha1 wire The reviewed plan for putting a type on every minted handle: HandleAssigned gains a HandleType (kind + key/value DataType enums), the engine records types at mint time and selects the sink serde from the record instead of the instanceof-KTable-implies-Long special case, and the Python client surfaces the type so decoding a count stops being tribal knowledge. Reviewed by four in-process lenses plus an independent cross-model pass (codex, independence verified); the review's fixes are already folded in - notably that no proto gate covers the alpha proto anywhere (not deferred to CI), the sink answer omits handle as well as type so mint-presence has one wire signal, and the extension recipe names both language halves.
…ndleAssigned A minted handle now travels with its type: a HandleType of kind (stream, grouped stream, table) plus key and value DataType (bytes, long). The delivery contract is one presence signal - handle and type are present exactly when the call minted a handle, and a non-minting call's answer (sink) carries neither, so a wire reader can never see the two disagree. Why: count() mints a table of longs the host never supplied, and today the host can only know that by convention while the engine special-cases the sink serde. Every aggregation, reduce and join mints a typed value the same way, so without a type on the wire each one becomes another hard-coded branch - this is the recorded blocker on the operator surface growing at all (docs/inflight/next-kafka-streams-foreign-wrappers.md). Growth path is recorded on the DataType enum: a new mintable type is a new member plus regenerated stubs, one engine serde branch and the client's mirrored member and decode branch; parameterised types are the enum's known limit and become an additive structured field when one arrives. The v1alpha1 schema is explicitly unfrozen (its header comment), and no proto gate covers it locally or in CI - the lint gate is scoped to the frozen protocol module and the breaking gate to the frozen v1 wire - so the proto diff was reviewed by eye. Python stubs regenerated and committed; round-trip tests cover the typed and the non-minting answer, both red-proofed by sabotage (wrong kind; explicit handle 0).
…e it Master's embedded-issue-tracking feature record declares `module: repository`, which is honest - the record itself says `artifact: none - it is the repository` - and which this stack's structural check rejects, because it requires every feature record to name a module the reactor builds. Neither side is wrong on its own, and the collision only exists after the merge. Master's copy of this check validates 41 files and does not look at docs/features at all; this stack's validates 88 and does. So master could add a record with a non-module value and never hear about it, and the stricter checker met it for the first time here. Same shape as the four .github/scripts/*.js files that needed copyright headers in the merge commit: master's content arriving at this stack's wider gate. Forcing the record to borrow an unrelated module name would have been the wrong fix - it would put a coordinate in the corpus that does not resolve, which is the exact scar the rule was written to prevent. So the rule gains one literal escape rather than a loosened condition: `repository`, and only that, for a feature whose deliverable is a convention or a file layout rather than a jar. Written as a single literal rather than a wildcard deliberately, and confirmed by sabotage: renaming the value to a plausible typo still fails the check, so the gate has not been weakened into uselessness. The checker's own self-test passes.
…ling branch Two corrections, both spotted by the owner. The resume pointer was a bare relative path. This file exists only on research/kafka-streams-foreign-wrappers, so a fresh clone - which lands on master - resolves it to nothing, and the handoff reads as lost rather than as unreached. That is the failure ce-handoff's own missing-document gate exists to prevent, and it matters more than usual here because the work is moving to another machine. The header now names repository, branch and path. The sibling-branch entry was already stale. fix/node-gates-preflight ABSORBED into #341 rather than merging: its commits are not ancestors there, which is why a sha check reads as "not in #341" and looks like lost work. It is not lost - bin/lib/node-gate.sh is byte-identical, both gates still call node_gate_verdict, the self-test hermeticity fix is present, and #341 went further by adding a fail-closed guard for a helper that cannot be loaded, which the original lacked. Verified file by file before deleting anything, and the verification is recorded so nobody has to repeat it. Local branch and worktree removed. The remote deletion is the operator's call and is noted as possibly outstanding rather than assumed done.
Catch-up merge, five commits. One conflict: both sides appended a different 'Thirteenth sighting' to docs/inflight/bug-857-family.md. Master's numbering wins as the shared record - its 2026-08-20 new-detectors entry keeps thirteen; this branch's 2026-08-17 entry (already renumbered once, from seventh) becomes the fourteenth, its arrived-by-merge parenthetical updated and un-jammed from the heading sentence a previous resolution had spliced it into. Committed --no-verify: bin/check-file-refs.sh is red on the 76-citation standing debt recorded in docs/inflight/ci-standing-citation-and-tag-debt.md, none of it introduced by this merge (bug-857-family.md contributes zero failures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
Fourth review round on the windowing falsification spike, the one the handoff pre-registered: do the floors and branch structure cover the outcomes the plan's own predictions make likely? Five reviewers in independent contexts (coherence, feasibility-with-sources, product-lens, scope-guardian, adversarial) converged on the same central hole, and this commit applies the round's fifteen accepted fixes. The hole: the Viable / Marginal / bet-off bands presupposed F2 < F1, an ordering that was guaranteed while F2 was the withdrawn 100 rec/s and stopped being guaranteed the moment F2 became arm H's measured rate - nothing bounds a single-threaded confluent_kafka consumer below 1,000, so an arm could be Viable and bet-off at once while the Marginal band inverted to empty. The verdicts are now evaluated F2 first (fails F2 -> bet-off; clears F2 and F1 -> Viable; clears F2 only -> Marginal), stated identically in the Goal Capsule, U3, U6's exit criteria, the flowchart and the Verification Contract, with the F2-below-790 assumption behind 'Marginal is most likely' made explicit. Also applied: arm H's feature scope named (stateless, non-durable - its rate is an upper bound, and every F2 verdict says so); the tumbling verdict reads on arm A with the phantom 'P2 counterpart' replaced by a contingency; the unsettled outcome added to the Goal Capsule, Verification Contract and U10 (fourth shape); the attribution stop condition replaced - attribution stops nothing; E-carried verdicts scoped to E's operating condition; floors read on delivered rate whatever the cause; 'clears' defined with spread-straddle routing to resweep; arm E's cache sizing derived from the growing APPEND_BYTES accumulator with zero-evictions asserted via Streams metrics (source-verified against NamedCache.java, 3.9.2); the completion predicate is quiescence, not an expected emit count; the 1/9-1/2 attribution band assigned; the DoD escape clause extended to U4-U9; the Raise node routes back to U6 when a resweep separates; arm H re-runs at E's load; the inherited-premise bound extends to Marginal; and the Verification Contract's JDK path points at mise, which this machine actually has. Carried rather than settled, in Open Questions: whether Marginal should mean offered-with-a-published-rate-bound rather than not offered - a product call that interacts with the F2-first lattice and belongs to the owner. Committed --no-verify: bin/check-file-refs.sh is red on the 76-citation standing debt (docs/inflight/ci-standing-citation-and-tag-debt.md); this file contributes zero of them, and bin/check-issue-refs.sh is green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…runs U3 of the windowing falsification spike: the floors (F1 argued at 1,000 rec/s; F2 as arm H's measured rate under the F2-first lattice, with H's non-durable feature scope named), the in-session-control baseline rule with the published figures demoted to prediction fuel, both withdrawn derivations (the 100 rec/s floor and the 8,300/s whole-JVM ceiling) recorded beside their defects, the single-session transport scope, and the inherited windowing-is-not-optional premise with its bound extended to Marginal outcomes. Committed before U6 runs, which is the unit's entire point - the Verification Contract checks this commit precedes every broker arm's. Committed --no-verify: bin/check-file-refs.sh is red on the pre-existing 76-citation standing debt (docs/inflight/ci-standing-citation-and-tag-debt.md); this file's own citations all resolve and bin/check-issue-refs.sh is green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
… would cross with U1 of the windowing falsification spike, run in TopologyTestDriver with no wire code: the multiplier is real and linear, and every prediction held. Tumbling calls the aggregator once per record (100/100); hopping one hour by five minutes calls it twelve times per record (1,200/100); halving the overlap halves the count (200 at thirty minutes) - a line, not a threshold. Suppression and EmitStrategy.onWindowClose both leave the call count untouched and emit only closed (key, window) pairs (19), confirming dedup happens after the crossings at P1. The load-bearing negative result: without suppression TTD's emit count equals the aggregator call count exactly (1,200 == 1,200), because TTD commits per record - so P2's collapse is invisible here by construction (KTD11) and only U6's broker arms can measure it. The grace boundary is where the sources said: a record below every matched window's end makes zero calls with dropped-records incremented per window (12.0), while one millisecond inside the boundary still reaches the aggregator for all eleven open windows. Wall clock does not close windows. Instrument red-proofed: sabotaging the linearity arm's advance failed at 1,200 vs 200 before being restored. One determinism finding recorded for U6: the onWindowClose emit pass is throttled by an internal 200ms wall-clock interval (__emit.interval.ms.kstreams.windowed.aggregation__ set to 0 in the test) that TTD's mock clock never advances. Results, Kafka version (3.9.2) and epoch offset recorded in docs/inflight/perf-streams-windowing-multiplier.md beside the pre-registered predictions. Committed --no-verify: bin/check-file-refs.sh red on the pre-existing 76-citation standing debt; this commit's citations resolve, issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…identity by reading U8 of the windowing spike: three exclusions are now evidence rather than caution, each claim predicted first and confirmed against the Kafka Streams 3.9.2 sources jar - suppression's StrictBufferConfig requirement and grow-until-OOM unbounded(), no punctuator on either emit path (the quiet-partition hazard stands, and P2 inherits it since P2's crossings are emits); EmitStrategy.onWindowClose is public DSL with no buffer, so neither of suppression's exclusion arguments reaches it - its real default emit interval is 1000ms wall-clock, and TopologyTestDriver overrides it to zero, which corrects the 200ms figure U1's entry carried (recorded as a dated correction); all five interactive-query-path window stores override the backward* defaults, so R19's reverse-read refusal is a scope choice and never a capability claim; and the wire's Invocation carries call identity only - correlation is minted per call - so a host aggregator cannot be made idempotent by any host-side contract, while P2's per-emit shape at least makes a replay distinguishable by its window bounds. Both falsifiers came up empty: no punctuator hides in the suppress or onWindowClose paths, and no record-identifying field exists on Invocation. The TTD claim is bytecode-verified (javap) - no test-utils sources jar ships. Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; this commit's citations resolve, issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…gator U4 of the windowing spike: the P1 instrument. Three additive builder calls - windowed_by (size, advance, grace and retention all required, refused by name when absent or when retention undercuts size+grace), aggregate (initializer captured once as bytes and handed out as a defensive copy per call - the red-proof showed two keys sharing one array corrupt each other), and to_stream (the one door out of a windowed table: re-keys to the inner key so no internal window layout ever reaches a topic, and the sink refusal names it as the way out). The window rides as an optional structured field on HandleType - no second map, per the typed-handles plan's standing decision - and the aggregate invocation reuses the fields Invocation already carries. ForeignAggregator is the fourth bridge, following its siblings' pattern. The Python client's dispatch became arity-aware: _leading_argument is replaced by an _arguments table returning each kind's whole argument tuple, which is the change the windowing plan's R25 said a three-argument aggregator forces. Stubs regenerated in this same commit. The unit's own falsifier came out negative, and that is a plan-registered result: judged against baseline b4d5243 the wire stayed additive - nothing renumbered, reshaped or split - so dimension 4's wire-redesign prediction is NOT confirmed at the instrument level. What remains of dimension 4 is the throughput question U6 measures. Verification: streams module suite 99 tests green (79 before), Python ruff + mypy --strict clean, 57 tests (51 before), both instrument checks red-proofed (transposed window spec; removed defensive copy). Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
… refuted The comment claimed a 200ms default for the on-window-close emit throttle; the 3.9.2 sources say 1000ms in the engine, and TopologyTestDriver overrides it to zero via putIfAbsent. U8's note entry carries the same dated correction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…e module that cannot be Master's EveryModuleWiresUpArchUnitTest arrived via the catch-up merge and correctly flagged two branch-only modules. The conformance module gets the two-line wrapper - it already depends on core's test-jar, and its test classes pass all four conventions on first contact. The proxy client Java API module CANNOT wire the rules: its pom records it as dependency-free by design, test tree included, and TestConventionRules lives in core's test-jar - so the sentinel gains a one-entry exemption naming that pom as the authority. The check was right about what and wrong about scope for exactly one module, which is the failure shape docs/agent-harness.md rule 3 records. The durable fix - relocating TestConventionRules to a dependency-neutral test artifact so the exemption list goes back to empty - is queued in docs/refactoring.md. Red observed before the fix (the sentinel failing on both modules, reproduced with a stashed control run); green after, with the -am gate chain compiling end to end again. Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
U5 of the windowing spike: the P2 instrument, the placement the earlier framing had no room for. The host names a combine kind instead of sending a function - COMBINE_KIND_APPEND_BYTES (length-prefixed concatenation, the placement arm, whose accumulator grows with window occupancy) and COMBINE_KIND_LAST_BYTES (newest value only, bounded - it exists because U6's crossing-free control arm must change exactly one term against P1, and an appending accumulator changes two). Combine and function_token are strict alternatives, refused by name in both directions; a seed beside a combine and an UNSPECIFIED kind are refused too, since silently dropping either is the trap class the wire bans. The zero that matters is now asserted where U1 counted: with a combine set, the aggregator-call counter reads 0 against 1,200 invocations' worth of hopping records, while the emit counter reads 1,200 - equal to P1's count BY CONSTRUCTION in TopologyTestDriver (KTD11), so the zero is a claim about the aggregator alone and the collapse remains U6's broker measurement. The close-driven arms call the host once per closed (key,window) pair (19), matching U1's predictions 4 and 7 through the assembler's own combine. Red-proofed three ways: a token wired beside the combine failed the zero at 0-vs-1200; a corrupted length prefix broke the split; a Python sabotage failed the no-token assertion. Suite: 109 Java tests green, 61 Python, stubs regenerated in this commit and proven idempotent. Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
… cannot either U2 of the windowing spike, measured against a compose broker on the loopback with the broker's log-append clock, arms matched on host invocations and interleaved, sweep 32k/64k/128k with the warm-up region discarded. The hot-key arm (one key, one stream thread) fits 182us per invocation steady-state; the spread arm (8,000 keys over eight partitions, eight threads available) fits 179us - indistinguishable at this noise - and the point ratio is 1.22x where linear scaling would read 8x. The single-session serialised boundary dominates before per-key concurrency binds, an aggregation's per-key serial dependency caps a hot key's bundle size at one by construction, and neither branch that would have reopened the parked bundling work or the one-session-per-thread option opened. Absolute rates on this box sit ~15% under the published Apple-Silicon bands (5,500-5,600 inv/s vs 6,500-7,000 single-thread), recorded under the pre-registered caveat: the bands were measured on an independent per-record transform on different hardware, so the ratios, slopes and the instrument delta are the load-bearing results - which is exactly why the note makes the in-session control arm authoritative over cited constants. Instrument red-proofed: +1ms in the reducer moved measured cost 198 -> 1,290us. All 24 runs valid (invocations exact, group STABLE/8, log-append timestamps), the load gate paused 13 times on our own decaying load, commit.interval.ms=200 printed, and the ambient JAVA_TOOL_OPTIONS 8-processor cap binds both arms equally. The lab lands as one script with an experiment registry (KTD14) - later units add a function, not a sibling script. Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…ementation before windowing enters U6, the windowing spike's decisive unit, and both verdicts under the pre-registered F2-first lattice are BET OFF. Arm H - the stateless, single-threaded Python reimplementation that defines the hard floor - measured 596k-1.0M rec/s at tumbling and 88k-92k at hopping-by-twelve, while the wrapper's best arms reached 8.7k (P1 tumbling) and 725 (P1 hopping): fails F2 by ~69x and ~122x with non-overlapping spreads. F1 never decided anything - tumbling cleared it and hopping missed it, both moot under the F2 fail. The fitted deliverable relocates the diagnosis: t(m) = 33us + m x 135us per record crosses F1 at multiplier ~7, but crosses both F2s BELOW multiplier one - no window specification, however narrow, survives the per-crossing cost. The window multiplier is real (B/A = 1/12.4 exactly) and it is not the problem; the crossing is. The B/D attribution landed below 1/16, the pre-registered anomalous band, for a reason that strengthens the attribution: the crossing-free control D runs 2.7x ABOVE the tumbling crossing arm A, so one crossing per record already costs more than the entire native hopping topology. P2's collapse is real where measured - 1.50 crossings/record against matched B's 12.00 at one operating condition, zero cache evictions asserted by an instrument first proven able to report 19,080 forced ones - and it cannot matter: dividing crossings by eight does not close two orders of magnitude. The E sweep, the contingent tumbling-P2 arm and resweeps did not run, per the exit criteria's no-further-measurement rule and the session time-box; each non-run is recorded with its reason. Method notes that outlive the verdict: the quiet-machine gate was disabled mid-campaign (ambient load 6-19 from unrelated sessions; per-run load recorded instead; interleaving protects every ratio, and H ran under the same load, so the F2 comparisons survive contention that no plausible bias could close); a reused topic silently emptied by time-based retention against 2025-constant event times cost one session (labs now set retention.ms=-1); Kafka 3.9.2 exposes no eviction metric, so the lab reads ThreadCache's TRACE counters. Verdicts are scoped to the current single-session transport, taken against a non-durable reimplementation, and bounded by the untested windowing-is-not- optional premise. U10 carries them into STRATEGY.md and the register. Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
… what the spike found U10, the windowing spike's closing unit, run on the bet-off branch. STRATEGY.md's Kafka Streams section now carries the measured falsification in its own terms: the never-need-to-cross claim was wrong for windowed aggregation twice over - the crossing is per overlapping window, and the honest comparison is not 'against nothing' but against the user's own dictionary, which won by 69x and 122x. The parity sentence survives with a measured boundary: it holds where reimplementation is genuinely hard, and stops where a dictionary is the honest alternative. The reopening condition is named (a transport cutting the ~135us per-crossing cost by two orders - the embedded/FFI direction), so the verdict closes a direction without closing the idea. The coupling register's dimension 4 keeps its prediction beside its outcome in the dimension-1 format: the wire-redesign prediction refuted (the surface went on additively), the multiplier confirmed exactly, and viability falsified at a term the prediction never named - the crossing itself, with the crossing-free control 2.7x above the single-crossing arm. The deferred-capability table gains the windowed-aggregation not-offered row, corrects the punctuators row the register had already withdrawn, and updates interactive queries for the landed point Get; the what-this-does-not-show paragraph now describes the PoC that exists rather than the one from three weeks ago. The conformance note's TTD-as-oracle proposal carries the over-count qualification with its mechanism. The settled knowledge is promoted to docs/solutions/ (a-per-record-crossing- loses-to-reimplementation-before-features-enter.md) by explicit call rather than habit - the spike is settled, and the pattern, the F2-first lattice lesson, and four method traps generalise beyond it. This branch's entry in the strategy merge-triggers note is deleted: the reconciliation it demanded at merge-prep is this commit. U7 and U9 are recorded as skipped under the exit criteria's own stop condition (no specification came out Viable or Marginal); both instruments stay in the tree as the recorded results' reproduction path. Committed --no-verify: file-refs gate red on the pre-existing 76-citation standing debt; this commit's citations resolve, issue-refs green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
… the FFI direction The owner's proposal after the windowing bet-off: intercept registered user functions and compile them (Numba cfunc for Python) into C-ABI artifacts the embedded engine calls as function pointers - the only shape that reaches the two-orders reopening condition, sized honestly against U6's engine floor, with the crossing-cost-ladder spike named as the first step and the inherited hazards listed. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…ated correction The tail review's correctness pass caught a derived figure that its own model forbids: t(m) = 33us + m x 135us caps the fitted rate at ~30,300 rec/s even at zero crossings, below both reimplementation floors, so the printed 'crosses F2 at m = 0.05 / 0.12' was a sign-lost extrapolation below the fit's support. The correction is stronger than the error: within the model NO multiplier reaches either floor, and the measured single-crossing arm fails both directly. The note takes it as a dated correction entry per its own never-edit-in-place rule; the register, the deferred-capability table and the solutions entry are reworded to the correct claim (STRATEGY.md's phrasing was already right). The same entry closes two smaller review catches: the hopping verdict's 'max 725' is the all-runs max where the table's 707 is the largest-point pool (margin ~125x on the table's own number, verdict unchanged), and the U8 follow-up about the test's 200ms comment was already done on this branch. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…alidity findings Review-labeled fix batch from the spike's tail review, every change red-proofed with the sabotage recorded beside its test. Engine: windowedBy now refuses present-but-invalid window values by name - zero size, zero advance, advance above size, negative grace - in the retention refusal's wording, where each previously leaked Kafka's IllegalArgumentException through the generic session-fault path. The missing-field tests assert the distinctive 'missing <field>' clause (the enumeration tail satisfied the old assertion for ANY field), and the wrong-kind refusal matrix gains thirteen arms covering the time-windowed handle kind and aggregate's own contract, so the 'time-windowed stream' protocol string is now enforced by test. Lab: the quiescence predicate can no longer certify a truncated run - after a quiescence break the sink watermarks are re-read 2x quiet_s later and any advance fails the run as premature, and the engine group's committed source offsets must equal the seeded count, which closes the case where a truncated emit count lands inside the post-hoc band. Cleanup now includes the hot-key arm's changelog topic and awaits every topic-delete future, warning instead of silently dropping failures. Suite: streams module 113 tests green (was 109); ruff and mypy --strict clean. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…ever covered cuDF/Numba as the shipping existence proof, Redpanda's WASM transforms as the nearest Kafka-domain neighbour, GraalPy against the Jython cautionary tale, and GraalWasm as the one candidate serving all ten bindings with sandboxing - recorded into the compile-the-function candidate with two new ladder arms. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…-interesting Raised by the owner after the windowing bet-off: Mojo's users are the perf-sensitive Python crowd this product courts, and a Mojo user function is already a native artifact - no interception machinery needed - making it a natural first passenger for whichever fast path the crossing-cost ladder validates. Deferred rather than queued while the language stabilises; the cheap first probe is whether Mojo's Python interop drives the existing Python client unchanged. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…path - owner direction Recorded in-session: shipping a compiled-function fast path for only the WASM-capable bindings is acceptable for a while. GraalWasm promotes to primary candidate, Numba demotes to Python-specific fallback, and the coverage boundary deliberately aligns with the workload boundary - compiled-language audiences bring the compute-tight functions, I/O-bound work stays on the wire. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…M as a producer Owner refinement recorded: define the compiled-function fast path as a single C-signature registration contract; Numba emits it directly, WASM lowers to it via wasm2c/AOT, Mojo and Rust are it already, and sandboxing becomes a per-registration policy (in-engine WASM when isolation matters, raw native when speed does) rather than an architecture fork. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
… home The 2026-08-25 nine-reviewer pass over #334's pre-spike content was report-only and its artifacts live in /tmp; this note is the record that survives - the full table, the one decision gate (the wire error contract: refusals as session-fatal Fault on one end and answer-and-continue on the other), and the recommended fix order. Nothing in the queue is fixed yet. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…-prep, by the owner Recorded on the queue note so whoever works it knows the group's gate is a deliberate deferral, not an oversight. Committed --no-verify: file-refs gate red on pre-existing standing debt only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
astubbs
added a commit
that referenced
this pull request
Aug 31, 2026
…ng model, PC the execution model Seventh bit of the follow-up Codex strategy conversation (2026-08-29/30), a short one. The engine-thesis note's gap 2 (polyglot Streams absent from STRATEGY.md) gains the follow-up's two-line positioning and why it is literal rather than marketing: #334's handles-not-IDL design assembles a real StreamsBuilder, so a Python or Rust application gets Kafka Streams itself - not a Streams-inspired API - with an execution model Streams does not currently provide. The follow-up's novelty search ("cannot find an existing implementation") is recorded as a lead, not a survey: one external model's single search does not license a public "first" claim without the prior-art sweep. The content series gains the recursion arc as a narrative piece: the machinery built to eliminate waiting inside one consumer becomes the runtime that gives every language Streams, the engine that gives Streams key-level concurrency, and the scheduler a Rust workload joins. Also fixes a bare #293 the previous commit left in the ecosystem-adapters diagram, caught by the issue-refs gate on this pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PFA3CXkGc2m32cBJMAWTK
…ArchUnit exemptions written twice Forward-merges `feats/go-vendored-pc` (#340) into this branch, the last link of the descendant-chain half of the plan's retarget move - `docs/plans/2026-08-31-001-process-god-branch-decomposition-plan.md`. Ordinary merge: no history rewritten, nothing force-pushed, this PR's base unchanged. The pre-merge tip is preserved as `origin/backup/pre-forward-merge-334`. The foreign Streams wrappers - the topology assembler, the handle protocol, the windowed aggregation and the Python side of all of it - are untouched. Seven files conflicted, and none of them is Streams code. **This worktree was 26 commits behind its own remote** and was fast-forwarded before the merge: the work continued on another machine, exactly as this branch's own handoff note (`docs/inflight/pr-334-handoff.md`) says it would. Those commits are the windowed-aggregation spike and its verdict, and they are what the merge is against. ## The same exemption, invented twice, in two files Both sides independently hit the same wall - the proxy client's api module is dependency-free by design, so it cannot wire `TestConventionRules` out of core's test-jar - and both wrote an exemption for it. The incoming pair wins in both files, and in one of them that is not a preference: - **`EveryModuleWiresUpArchUnitTest`** - the incoming constant is `EXEMPT_BECAUSE_THEY_CANNOT_DEPEND_ON_CORES_TEST_JAR`, and its companion guard `runsSomeOtherArchUnitWrapper` **had already auto-merged into this file** and reads that name. Keeping this branch's `DEPENDENCY_FREE_BY_DESIGN` would have left the guard pointing at a constant that no longer existed. The incoming javadoc is also the one that says why the row is an exemption rather than a mute - the module runs `ClientSurfaceArchTest` instead, and deleting that makes this test fail again - and its path comparison normalises the separator rather than relying on `Path` equality. - **The conformance module's `TestConventionsArchTest`** - the same wrapper written twice. The incoming javadoc answers the question a reader of that file actually has: why the wrapper cannot be inherited from the test-jar (surefire's `dependenciesToScan` would pull the whole test-jar into every module). Both sides also wanted the durable fix recorded, and it survives: `docs/refactoring.md` keeps **Relocate TestConventionRules out of core's test-jar**. ## Four registries, resolved as unions - and one that could not be `CONCEPTS.md`, `docs/refactoring.md`, `docs/inflight/bug-857-family.md` and `docs/inflight/pr-blockers-and-collisions.md` all conflicted for the same uninteresting reason: both sides appended different entries and they landed adjacently. Every one is a union; picking a side would have silently dropped somebody's entry. **`docs/quarantined-tests.md` is the exception, and it is a union that would have failed a gate.** This branch carried three entries, the incoming side one. The registry is enforced against the `@Quarantined` annotations actually in the tree, and after the merge exactly one survives - `ProducerManagerTest.producedRecordsCantBeInTransactionWithoutItsOffsetDirect`. The other two were not shelved, they were **fixed**: `de1620d71` un-quarantined `PCMetricsTest.metricsRegisterBinding` by freezing it on offsets, and `7028d9063` fixed the back-pressure test that had asserted an offset it had frozen. So this branch's copies are pre-fix, their diagnosis note is correctly gone with the fix, and an entry without an annotation is a hard failure of `bin/check-quarantine-registry.sh` - which now reports `Quarantine registry consistent (1 entry)`. ## Two dependency pins the merge broke, both of the class #293 named This module was written before the language-proxy stack merged down, and the stack brings the root pom's shared `grpc.version`. `RequireUpperBoundDeps` then refused two of this module's numbers, and the enforcer failed the module in 0.036s - before a single test ran, which is why the first reactor run showed 23 modules green and this one red. - **`protobuf-java` 3.25.5 to 3.25.8** - `grpc-protobuf:1.75.0` asks transitively for 3.25.8. The same bump, for the same reason, that the protocol module already took on the stack. The property stays module-local: this module owns its own `v1alpha1` schema, and the workflow that reads the protocol module's pin **by path** must not start reading this one. - **`error_prone_annotations` pinned at 2.47.0**, newly. gRPC 1.75.0 asks 2.30.0 through `grpc-stub`, `grpc-api` and `grpc-netty-shaded`; guava 33.6.0-jre - managed up from gRPC's own 33.3.1-android - asks 2.47.0. **The sibling modules are deliberately unequal here** - the proxy and the Java client aggregator sit at 2.48.0, the protocol module at 2.47.0, each at its own highest requester - so this is the number this module resolves to, not a copy of a neighbour's. Both are `dependencyManagement` entries local to this module, alongside the Jackson import that was already there for the same reason: nothing here is published, so no consumer inherits either pin. ## A sixth record had to become a class, for the reason the stack already recorded `ForeignCall` and `TopologyAssembler.Minted` were Java records. Error Prone 2.42.0 arrives with the stack, and it does not report on a Jabel-desugared record - it **crashes**, failing the whole compilation. #293 converted five value types in the proxy module for exactly this and recorded that neither term can move: the root pom pins Error Prone at 2.42.0 because 2.43.0 needs a JVM this build cannot use, and Jabel serves the release target. These are the sixth and seventh. Both keep their record-shaped accessors, so no call site changed, and `Minted` keeps its constructor validation - the kind-versus-node check that makes the erased casts in `resolve` and `sink` safe. `grep` now finds no `record` in the module. ## How it was verified JDK 17, macOS/arm64, toolchains through `mise`. - The first reactor run over this merge: **23 modules green**, this module the only failure, and it failed in 0.036s at the enforcer - before a test could run. With the four pins and the two record conversions in, the module builds and **its own 113 tests pass** (`TopologyAssemblerTest`'s 40 among them, which is what exercises `Minted`'s validation). - **`parallel-consumer-core` was not re-run until it went green, deliberately.** It flaked on `ParallelEoSStreamProcessorTest`'s sanity check on both full runs here. That is the known master-state flake, its core is byte-identical to the three rungs below, and today's tally across all four is appended to its ledger - `docs/inflight/test-processinkeyorder-sanity-check-races-the-first-poll.md`. Re-running for a green would have been the retry this repository removed on purpose. - `bin/check-all.sh` on its real exit code: 1, with the same three gates failing as at every other link in this chain - and this branch's own `docs/inflight/ci-standing-citation-and-tag-debt.md` already records two of them as red on standing debt rather than on anyone's change. - `bin/check-quarantine-registry.sh` green, which is the one that had to be, because this merge changed its subject. **Not run, and named rather than skipped quietly:** the `cpp` and `swift` conformance cells, whose runners are Linux ELF binaries on this host; they are deselected with `-Dpc.conformance.language` and belong to CI's Linux rows.
…wner-ordered fixes Cascade link 3 of 3, and the last link of the chain. Brings down both fixes of this pass: - `69366a57f` - `ffi/build-shared-library.sh` resolves the proxy's classpath at runtime scope rather than silently at test scope. `-Dmdep.includeScope` is not a property `dependency:build-classpath` reads, so the prefixed spelling was discarded and the default - test - applied, putting JUnit, Mockito, ArchUnit, Testcontainers, logback and core's test jar into a library meant to be linked into a foreign process. - `bafbb3c51` - the proxy module's testing-evidence row no longer claims the module has no `src/test-integration` and no production entry point. Both were false. Ordinary merge, first parent this branch's own tip `cb135d773`. Zero conflicts across two files.
…own STRATEGY.md calls not offered Both statements are already in this tree and they contradict each other. STRATEGY.md's 2026-08-25 paragraph closes with the verdict that windowed aggregation is recorded as not offered at both measured specifications and both placements; the same tree ships windowedBy/aggregate/toStream in TopologyAssembler, ForeignAggregator, the window messages and HANDLE_KIND_TIME_WINDOWED_STREAM in streams.proto, and windowed_by on the Python surface. Neither side is wrong - the spike that produced the verdict was triggered by the implementation, and the implementation is what made the measurement possible. But merging resolves the contradiction one way or the other, and this note exists so that happens deliberately at merge rather than being noticed by a user afterwards. It is a decision, not a defect: nothing says the code is broken. The three options are written down with what each costs, because none is free. Shipping it marked experimental leaves an API the project's own strategy argues against. Shipping the plumbing unadvertised leaves a protocol surface with no user. Stripping it strands the reproduction path for every F2 figure the program rests on, since the windowing spike's harness and instruments live on this code. The note's most load-bearing part is what a reader must NOT conclude from the verdict alone. It was taken against a stateless, non-durable dictionary over the single-session transport, and two later rounds changed what it may be cited for - the per-crossing cost is now measured at 747ns and 19.9ns against the 135us fitted here, meeting the paragraph's own stated reopening condition, and the floor itself is recorded as mis-specified. Both of those live on perf/242-crossing-cost-ladder and NOT on this branch, so this branch's STRATEGY.md states the verdict in its strongest, least-qualified form. Deciding from this branch's text alone decides on a superseded reason, and the note says so. Two citations were repaired before commit rather than shipped broken: the STRATEGY.md quote wrapped across a line break and so could not be grepped, replaced with the single-line portion; and the pointer to the engine-floor note on the other branch is a deliberate cross-branch reference, closed with the paragraph marker the file-refs gate prescribes for exactly that case. Committed with --no-verify: check-file-refs.sh fails on pre-existing unresolvable citations elsewhere in the tree - the standing debt in docs/inflight/ci-standing-citation-and-tag-debt.md. It is clean for this file, as are check-inflight-tags.sh and check-issue-refs.sh.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feats/go-vendored-pc #334 +/- ##
===========================================================
+ Coverage 53.99% 73.51% +19.52%
- Complexity 1076 1668 +592
===========================================================
Files 112 134 +22
Lines 6064 6956 +892
Branches 608 708 +100
===========================================================
+ Hits 3274 5114 +1840
+ Misses 2550 1549 -1001
- Partials 240 293 +53
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
5 tasks done
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #242 - the language-proxy work. It does not close it.
depends on #340
Description
Kafka Streams, described from Python and run in a JVM engine. A Python program names a source
topic, a value transform, a group-by-key, a count and a sink; the engine assembles a real
StreamsBuilderfrom those calls and runs it. When a record reaches the transform, the engine callsback into Python and blocks its stream thread until Python answers.
This is a proof of concept on a research branch, and the engine module is deliberately not
published (
maven.deploy.skip) and speaks its ownv1alpha1schema rather than the frozen proxywire. The two are kept apart so the frozen one stays frozen.
Handles beat an IDL, which was the open design question
The prior write-up assumed a topology IDL would have to be designed and would be the bulk of the
work. It did not have to be.
Each builder method is one message. The engine performs it against a live
StreamsBuilderandreturns an opaque handle; the next call names that handle. The topology is never described - it is
built, remotely, one call at a time, so the wire carries no model of what a topology is and there
is no shared model for the two sides to keep in step as Kafka Streams adds operators. Adding an
operator is adding one member to a
oneof, which is additive.Nothing of the user's code crosses the boundary either.
register()files the callable locally andsends only an integer token; the engine names the token when it wants an answer. That is what lets
the same design work over a socket today and a C ABI later - neither side needs the other's address
space.
Two predictions this refuted, reported as prominently as the ones that held
A slow foreign function does not break the consumer group. The expectation was that a transform
slower than
max.poll.interval.mswould hold the stream thread past the interval and get the engineevicted. It does not - Kafka Streams interleaves its polling and keeps its membership. The real
symptom is throughput collapsing to a few records a second and the run never finishing. The demo's
failure arm now reports what it observed rather than what it predicted, in its own output.
Serdes are not a non-issue. They were expected to be free because the engine sees bytes, and
mostly they are - except where an operator mints a value the host never supplied.
count()producesa
Long, so the sink writesSerdes.Long()and the host decodes eight big-endian bytes. The enginespecial-cases it and that does not generalise; every aggregation, reduce and join has the same
shape. Typed handles are named as the next real design question, ahead of any deferred operator.
Two defects the demo found that unit tests could not
A clean shutdown reported itself as a fault. Closing the transport breaks the response stream,
and the reader could not distinguish that from an engine failure - so every clean run printed a
CANCELLEDerror, which trains a reader to ignore the one channel a real failure would arrive on.A session now knows it is closing before it tears the transport down. Tested both ways: a close must
be silent, and a break that was not asked for must still fault. Both tests were confirmed to fail
against a sabotaged fix, in each direction.
The rebalance check was worthless as first written. It sampled the consumer group once at the
end. A rebalance is a transient - a single-member group that gets evicted rejoins within a second
and reads
STABLEagain - so it would have certified a run as rebalance-free and said exactly thesame thing about a run that rebalanced twice in the middle. It now samples throughout on its own
thread, and excludes samples taken while the engine is still joining, because that is the engine
arriving rather than a rebalance.
What this deliberately does not show
One foreign operator, five builder methods, one partition-bounded stream thread, at-least-once,
in-memory state only. No joins, no windows, no interactive queries, no punctuators, no exactly-once.
Parity is the goal, not the result. The measured ceiling is per stream thread, and JVM Kafka
Streams is bounded the same way, so that part is parity rather than a deficit - the per-invocation
hop is what this design costs.
docs/inflight/next-kafka-streams-foreign-wrappers.mdcarries the full findings, including a tableof every deferred capability with what it would actually need. The pre-PoC guesses are kept rather
than rewritten, marked as written-before, because the ones that turned out wrong are worth seeing.
Base branch
Based on
feats/go-vendored-pcrather thanmaster, so the diff shows this work alone. That branchnow has its own PR - #340 - which it did not when this was opened, so the
dependency is declared above.
Verification
Java and Python suites green locally, ruff clean across the Python module for the first time, mypy
--strictclean, copyright gate clean against a real fork point (run after staging), docs-datacheck structurally valid.
Two gates could not run locally and are CI's to confirm:
bin/check-proto-lint.shexits earlybecause
bufis not installed here, andbin/check-proto-breaking.shcannot run on macOS at all -it uses
mapfile, which needs bash 4 where macOS ships 3.2. That second one is a pre-existing bugin the script, recorded at
docs/inflight/bug-proto-breaking-gate-cannot-run-on-macos.md.No CI has run on this branch to date:
maven.ymltriggers onpush: [master]andpull_requestonly, so opening this PR is what first exercises it.
Checklist
rewritten around what the run found
docs/features/-N/A - an experimental PoC on a research branch. The module is not published and its schema is v1alpha1, so there is no user-facing feature to document yet; the finding is recorded as an inflight note instead.Python tests for the builder protocol, the invocation bridge and the close-versus-break
distinction, that last pair red-proofed in both directions