#4515 Phase 2: binary event serialization on Quick + BulkEventAppender#4584
Merged
Conversation
#4578 shipped the foundation with `Rich` mode only; the Quick paths (QuickWithServerTimestamps default + Quick + bulk COPY) all guarded against binary events at store-build time. This lifts that constraint — binary serialization now works on every EventAppendMode and through the BulkEventAppender. ## Wire-format change: `mt_quick_append_events` grows a `bdatas bytea[]` param The PostgreSQL function used by both Quick variants now accepts a parallel `bdatas bytea[]` parameter right after `bodies jsonb[]`. The INSERT writes `bdatas[index]` into `mt_events.bdata`. For JSON events the array slot is NULL; for binary events `bodies[index]` is the `{}` placeholder and `bdatas[index]` carries the real payload. Same on-disk row shape as the Rich path: `bdata IS NULL` remains the discriminator that the existing read path keys off. Weasel's standard function-diff migration handles the signature change as DROP + CREATE on existing installations; existing JSON rows are untouched. ## Call-site dispatch — same shape as Rich `PostgresEventStoreDialect.BuildQuickDescriptor` and `BuildQuickWithServerTimestampsDescriptor` install the same `serializeEventData` / `serializeEventBdata` closures the Rich descriptor uses (look up `EventMapping`, branch on `IsBinary`). `QuickAppendEventsOperationBase.writeBasicParameters` now accepts an optional `Func<IEvent, byte[]?> serializeEventBdata` and binds the parallel `bdatas bytea[]` array. ## BulkEventAppender — bdata in the COPY column list `buildEventColumns()` adds `bdata` right after `data`; `writeEventRow` looks up the EventMapping per event and writes either the binary payload (for `[BinaryEvent]` types) or NULL (for JSON). The COPY format already supports NULL values per column, so no schema relaxation is needed. ## Removed: AssertNoBinaryEventsForQuickMode The Phase 1 guard in `PostgresEventStoreDialect` that threw at store-build time if a binary event type was registered with a non-Rich AppendMode is gone — no longer needed. ## Tests Three new tests in `QuickModeBinaryEventTests` (separate fixture so each test can dial in its own AppendMode): - `quick_with_server_timestamps_round_trips_binary_events` — mixed binary + JSON stream on the default mode, round-trip via the PG function. - `quick_mode_round_trips_binary_events` — explicit `Quick` mode. - `quick_mode_binary_events_land_in_bdata_column` — on-disk shape verification: binary rows have `data = '{}'` + `bdata != NULL`; JSON rows have `data = real JSON` + `bdata = NULL`. Regression checks: - Full Marten.MemoryPack.Tests suite: 8/8 ✅ - EventSourcingTests.end_to_end_event_capture_and_fetching: 83/83 ✅ - DaemonTests.Bug_3059_double_application: 1/1 ✅ (re-running the test that flushed out the column-count bug in PR #4578's first CI run) ## Docs `docs/events/binary-serialization.md` updated: - Removed the "EventAppendMode.Rich only" + "No bulk appender support" constraints from the Constraints section. - Added a new "Append modes" section explaining the feature works across all three modes + BulkEventAppender. - Quick-start example no longer forces `AppendMode = Rich`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… fix Weasel 9.0.2 (JasperFx/weasel#299) fixes PostgresqlMigrator.executeWithConcurrencyRetryAsync so it reopens a Closed/Broken connection before the retry attempt. That eliminates the recurring "Connection is not open" failure on the conjoined `EventSourcingTests.end_to_end_event_capture_and_fetching_the_stream. query_before_saving` test that hit this PR + #4576 + #4578 + #4582. Bumps Weasel.Postgresql + Weasel.EntityFrameworkCore 9.0.1 → 9.0.2 in Directory.Packages.props (CPM). Marten.MemoryPack.Tests still 8/8 locally on top of the new Weasel. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes #4579 with a docs-only answer rather than building a binary-side upcaster framework. The JSON upcasters (Marten.Services.Json.Transformations) operate on the JSON wire form and don't generalize to a byte[] payload; designing a typed transform shape for binary events is non-trivial and the use case can be addressed end-to-end today by leaning on Marten's existing per-event-type registry. The recommendation: introduce a new event type for each schema change (e.g. TripStarted -> TripStartedV2), have the aggregate handle both versions, and let the coexistence design carry old rows + new rows on the same stream without migration. The only caveat is that MemoryPack's in-place backward-compatible field evolution works for additive-only changes too, but stops at the serializer's tolerance rules (renames, type changes, splits) — versioned event types work for every shape of change and stay explicit. Replaces the "No upcaster support" constraint section with a "Schema evolution — use versioned event types" section that gives the recommended pattern with code samples + a sub-section on the "why-not-in-place" tradeoff + a note on mixing binary + JSON. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 28, 2026
This was referenced May 29, 2026
Merged
This was referenced Jun 8, 2026
This was referenced Jun 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 follow-up to #4578. Lifts the Rich-only constraint — binary event serialization now works on every
EventAppendModeand through theBulkEventAppender.TL;DR
The
AssertNoBinaryEventsForQuickModeguard from #4578 is gone — no longer needed.Wire-format change:
mt_quick_append_eventsgrows abdatas bytea[]parameterThe PG function used by both Quick variants now accepts a parallel
bdatas bytea[]parameter right afterbodies jsonb[]. The INSERT writesbdatas[index]intomt_events.bdata. JSON events get NULL in that slot; binary events get the bytes + a{}placeholder inbodies. Same on-disk row shape as the Rich path —bdata IS NULLremains the discriminator the read path keys off.Weasel's standard function-diff migration handles the signature change as DROP + CREATE on existing installations. Existing JSON rows are untouched.
Call-site dispatch — symmetric with Rich
PostgresEventStoreDialect.BuildQuickDescriptorandBuildQuickWithServerTimestampsDescriptornow install the sameserializeEventData/serializeEventBdataclosures the Rich descriptor uses (look upEventMapping, branch onIsBinary).QuickAppendEventsOperationBase.writeBasicParametersaccepts aFunc<IEvent, byte[]?> serializeEventBdataand binds the parallelbdatas bytea[]array right afterbodies.BulkEventAppender —
bdatain the COPY column listbuildEventColumns()addsbdataright afterdata;writeEventRowlooks up the EventMapping per event and writes either the binary payload (for[BinaryEvent]types) or NULL (for JSON). COPY natively supports NULL per column — no schema relaxation.Tests
Three new tests in
QuickModeBinaryEventTests(one fixture per AppendMode):quick_with_server_timestamps_round_trips_binary_eventsquick_mode_round_trips_binary_eventsQuickmodequick_mode_binary_events_land_in_bdata_columndata = '{}'+bdata != NULL; JSON rows havedata = real JSON+bdata = NULLRegression checks:
Marten.MemoryPack.Tests: 8/8 ✅EventSourcingTests.end_to_end_event_capture_and_fetching: 83/83 ✅DaemonTests.Bug_3059_double_application: 1/1 ✅ (re-running the test that surfaced the column-count bug in Foundation for binary event serialization (#4515 — Phase 1, Rich-mode only) #4578's first CI run)Docs
docs/events/binary-serialization.mdupdated:AppendMode = Rich.What's still deferred
Just binary event upcasters/downcasters — tracked at #4579. 4 open design questions in the issue body; waiting for a real consumer.
🤖 Generated with Claude Code