fix(#3233): a content file over the per-delivery budget travels out of band by rbuergi · Pull Request #3286 · Systemorph/MeshWeaver

rbuergi · 2026-09-04T07:13:27Z

The residual, re-measured

A GitSync content sync carries a Space's content/** binaries inline. #3097/#2885 partitions the
write so a delivery is ≤ budget + largest single file — which fixed every Space whose aggregate
was the problem. A file is the atom the receiving handler writes and is never split, so a file
whose packaged (base64) cost alone exceeds ContentDeliveryBudget.BudgetBytes (= Orleans'
MemoryStreamBlockBytes, 1,048,576, hard-coded with no configuration surface) travelled whole and
was refused wherever the Orleans transport was in the path.

Re-ran the issue's own cost function on Systemorph/MeshWeaver.Education@f7ae723 (2026-09-04) —
unchanged, 25 files over budget across all 7 Spaces:

Space	files	over budget	total	largest packaged
AgenticEngineering	25	12	101.2 MB	13,188,871
AgenticBusiness	9	4	27.2 MB	10,910,243
AgenticPrimerDe	7	3	10.9 MB	4,291,888
AgenticPrimer	7	3	9.8 MB	3,873,652
DataModeling	3	1	8.6 MB	10,929,144
AdvancedBusinessRules	2	1	9.5 MB	12,540,448
AgenticOffice	3	1	8.2 MB	10,224,527

The axis is "has a video", not "is large" — AdvancedBusinessRules is one of the smallest Spaces
and carries the second-largest single file.

The design

Out-of-band asset transfer. A file over the budget no longer rides the message: its bytes are
written into the destination collection's reserved _staging/ folder before the request is
posted, and the delivery carries a content-addressed StagedContentFile(Path, Handle, Length)
handle. The receiver streams the blob into the destination path exactly where the inline file would
have gone.

Where the bytes land — the destination collection itself. The producer reaches it the way
MeshOperations.Upload and the /api/content route already do: it resolves the collection's
config from the owning node's hub (a few hundred bytes) and writes through it locally. The
content store is already required to be reachable from a hub other than the collection's owner —
that is what the content route does — so this adds no assumption, and needs no well-known
collection name, no deployment key and nothing new to provision. A mesh-level staging store would
have needed all three for the same physical bytes on the same share.

Lifetime. The producer owns every blob it stages and deletes them in the post chain, after
the last delivery is answered (which is exactly when nothing can still name a handle) and before the
caller's answer — so "this sync answered" means "its staging area is clean", not a race a caller has
to poll. Only a crashed producer leaves residue; that is reclaimed by an age sweep (24 h) that runs
before staging and therefore cannot see a live transfer. The mirror never prunes _staging/
(excluded by name — it is framework state) and never prunes a staged file (its path is in the full
MirrorKeepPaths keep set, now mandatory whenever anything is staged, not only when the write is
split). Without both, the prune — which rides the first delivery — would delete the blobs the
following deliveries still have to read.

Idempotence. The handle is the SHA-256 of the bytes. Two identical files stage once; a re-run
addresses the same blob at the same key and the same file at the same destination path, so nothing is
duplicated, and a blob already present with the right length is not rewritten.

Failure behaviour stays honest — #3101 is not undone

A handle the receiver cannot resolve, or whose length does not match, fails by name with the
handle and the path. Never "zero files"; never a truncated or empty asset.
When staging cannot run (a store the producer cannot reach), the file falls back to inline —
today's behaviour, including today's refusal — and the failure names both halves: the A Space whose content sync is refused stops syncing SILENTLY — AgenticEngineering's assets are over the frame limit outright #3101
budget measurement and why the out-of-band road was unavailable.
ContentDeliveryBudget.DescribeOverBudget is now measured on the files that actually travelled
inline, so a file that went out of band is not reported as an over-budget payload nobody built.
StaticRepoImporter is untouched: a failed transfer still lands Warning with its reason on the
Space's _Activity/content-sync ledger, and the fingerprint still does not stamp Succeeded.

What did not change

The budget is untouched, the inline partitioning of #2885 is untouched, a file is still never split
(no chunking, no reassembly, no partial-write story), and authorization is unchanged — the content
write still happens on the owning node's hub under [SyncContentFilesPermission], and nothing lands
in the collection before that check runs. A sync with no over-budget file produces a request
byte-for-byte identical to before (no config round-trip, no staging pass, StagedFiles null).

Tests — and what they do without the implementation

test/MeshWeaver.ContentCollections.Test/OversizedContentTravelsOutOfBandTest.cs, driven through the
real SyncContentFilesRequest handler over a real file-system collection (no mocking, no
Task.Delay):

A_file_over_the_budget_travels_out_of_band_and_arrives_intact — the file arrives byte-for-byte
(SHA-256 compared), its bytes are on no delivery, exactly one staged handle equals the content
hash, every delivery fits the budget outright (no "plus one file" allowance), staging is clean.
A_rerun_duplicates_nothing_and_leaves_no_residue — two runs, one folder, both runs address the
same handle, no residue.
The_mirror_keeps_the_out_of_band_file_prunes_the_stale_and_preserves_an_upload — the transfer
survives the mirror in the same operation, the retired source file is pruned, the user upload
(Silent loss of uploaded collection files after hub teardown; mesh search returns zero results on deployed portal #435) survives, one prune pass, the staged path is in MirrorKeepPaths.
Staging_that_cannot_run_falls_back_inline_and_the_failure_names_both_halves.
A_handle_nothing_staged_fails_by_name_and_writes_nothing.

Revert experiment (disable the staging decision, rebuild src/ and the test project):
3 of 5 fail, with exactly the pre-fix evidence —
Did not expect collection {…, "videos/module1-intro.mp4", …} to contain "videos/module1-intro.mp4",
Expected collection to have 2 item(s) … but found 0, and the #3101 sentence with the #3233 half
missing. Restored: 27/27 in that project, 9/9 on the Graph content-sync tests, 250/250 on
the Documentation guard suite, DocumentationLinkIntegrityTest green.

Built -c Release -warnaserror, one project per invocation, 0 Error(s) / 0 Warning(s):
MeshWeaver.Mesh.Contract, MeshWeaver.ContentCollections, MeshWeaver.Documentation,
MeshWeaver.Graph, MeshWeaver.PluginCatalog, MeshWeaver.Mesh.Operations,
MeshWeaver.Hosting.AspNetCore, MeshWeaver.Connection.Orleans, and both test projects.

Docs

New: Doc/Architecture/OutOfBandContentTransfer — the design, the lifetime rules, the
idempotence argument, and the failure contract.
Doc/Architecture/OversizedDeliveryRefusal — its own "a single file larger than the budget travels
whole" residual is now closed and points here.
Doc/Architecture/ContentSyncVisibility — records that its observability contract is what the new
transfer's fallback rules preserve.
What's New (Category: Fix): Course videos sync again.

No public type or member is removed — additions only — so no Pairs-with: counterpart is needed.

🤖 Generated with Claude Code

…f band #2885 partitions a content sync so a delivery is `<= budget + largest single file`, which fixed every Space whose AGGREGATE was the problem. A file is the atom the receiving handler writes and is never split, so a file whose packaged (base64) cost ALONE exceeds the budget travelled whole and was refused wherever the Orleans transport was in the path. Re-measured 2026-09-04 on MeshWeaver.Education@f7ae723: still 25 files across all 7 Spaces, unchanged. The budget is Orleans' memory-stream block size, hard-coded with no configuration surface, and raising it is the move OversizedDeliveryRefusal exists to forbid. Such a file now goes AROUND the message. Its bytes are written into the destination collection's reserved `_staging/` folder before the request is posted, and the delivery carries a content-addressed StagedContentFile handle; the receiver streams the blob into its destination path. Where the bytes land: the destination collection itself, reached by resolving its CONFIG from the owning node's hub (a few hundred bytes) — the mechanism MeshOperations.Upload and the /api/content route already use. The content store is already required to be reachable from a hub other than the collection's owner, so this adds no assumption, no well-known name, no deployment key and nothing to provision. Lifetime: the producer owns every blob it stages and deletes them IN the post chain, so the answer means "the staging area is clean"; only a crashed producer leaves residue, reclaimed by an age sweep (24 h) that runs before staging and therefore cannot see a live transfer. The mirror never prunes `_staging/` (excluded by name) and never prunes a staged file (its path is in the full MirrorKeepPaths keep set) — without both, the prune, which rides the FIRST delivery, would delete the blobs the following deliveries still have to read. Idempotence: the handle is the SHA-256 of the bytes. Two identical files stage once; a re-run addresses the same blob and the same destination path, so nothing is duplicated and an already-present blob is not rewritten. Failure stays honest (#3101 is not undone): - a handle the receiver cannot resolve, or whose length does not match, FAILS by name and writes nothing — never "zero files", never a truncated asset; - when staging cannot run, the file falls back to inline (a monolith carries it fine) and the refusal names BOTH halves: the #3101 budget measurement over the files that actually travelled inline, plus why the out-of-band road was unavailable; - DescribeOverBudget is now measured on the INLINE remainder, so a file that went out of band is not reported as an over-budget payload nobody built. Tests (MeshWeaver.ContentCollections.Test, real handler + real FS collection): the over-budget file arrives byte-for-byte and is on NO delivery; every delivery fits the budget outright; a re-run duplicates nothing and both runs address one handle; the mirror keeps it, prunes the stale, preserves a user upload; staging that cannot run reports both halves; an unresolvable handle fails by name and writes no empty file. Reverting the staging decision fails 3 of the 5. Design committed as Doc/Architecture/OutOfBandContentTransfer; OversizedDeliveryRefusal's residual and ContentSyncVisibility updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…owed The reclaim's per-blob Catch continued past a failed delete — correct, so one stubborn blob cannot strand the rest or fail a sync that succeeded — but it swallowed the exception with nothing written down, which is exactly the silent-swallow shape AGENTS.md forbids. What it leaves behind is a blob on the content share that only the age sweep will now collect, so it says so. Both reclaim paths now take the builder's logger: Discard warns per blob it could not delete, and SweepStale logs what it reclaimed (Information — crash residue is rare and worth a line), warns on a blob it could not, and reports at Debug when the staging folder cannot be enumerated at all. The outer catch in Reclaiming stays as the defensive arm it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Two correctness/operability issues remain in the new staged-write and reclaim paths (unknown-size GetContentSize == -1 handling, and silent reclaim failures) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes content sync failures for single binary assets whose base64-packaged size exceeds the per-delivery payload budget by transferring those oversized files out-of-band through the destination content store’s reserved staging folder (_staging/), while keeping existing delivery chunking/mirroring semantics for the remaining inline files.

Changes:

Add a content-addressed staged-file handle (StagedContentFile) and plumb it through SyncContentFilesRequest as StagedFiles.
Implement producer-side staging + receiver-side streamed write for oversized files, and ensure mirror pruning keeps staged destinations and never touches _staging/.
Add end-to-end tests plus architecture/docs/What’s New updates describing the design, lifecycle, and failure contract.

File summaries

File	Description
test/MeshWeaver.ContentCollections.Test/OversizedContentTravelsOutOfBandTest.cs	New integration tests covering out-of-band transfer, idempotence, mirror/prune behavior, and failure reporting.
src/MeshWeaver.Mesh.Contract/MeshExtensions.cs	Registers `StagedContentFile` in the type registry for correct typed deserialization across hubs.
src/MeshWeaver.Mesh.Contract/ImportDeleteRequests.cs	Introduces `StagedContentFile` and adds `StagedFiles` to `SyncContentFilesRequest`.
src/MeshWeaver.ContentCollections/ContentStaging.cs	Adds staging folder constants and producer-side staging/sweep/reclaim helpers.
src/MeshWeaver.ContentCollections/ContentImportExtensions.cs	Adds receiver-side staged-handle resolution + write path, updates mirroring keep-set and prune exclusions, and stages oversized files before posting deliveries.
src/MeshWeaver.ContentCollections/ContentDeliveryBudget.cs	Updates budget documentation and clarifies that over-budget measurement applies only to files that actually traveled inline.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-04-course-videos-sync-again.md	What’s New entry describing the user-visible fix.
src/MeshWeaver.Documentation/Data/Architecture/OversizedDeliveryRefusal.md	Updates prior “residual” section to point to out-of-band transfer as the closure.
src/MeshWeaver.Documentation/Data/Architecture/OutOfBandContentTransfer.md	New architecture page documenting the design, lifecycle, idempotence, and failure behavior.
src/MeshWeaver.Documentation/Data/Architecture/ContentSyncVisibility.md	Updates visibility doc to reflect the new transfer path and preserved honesty contract.
src/MeshWeaver.Documentation/Data/Architecture.md	Adds the new architecture page to the Reliability & wedges index.

Review details

Files reviewed: 11/11 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-04T07:38:30Z

+        return target.GetContentSize(blob)
+            .SelectMany(size => size switch
+            {
+                null => Observable.Throw<int>(new InvalidOperationException(
+                    $"Staged content '{staged.Handle}' for '{staged.Path}' is not in collection "
+                    + $"'{target.Collection}' staging area ('{blob}') — the out-of-band transfer "
+                    + "did not complete, so the file was NOT written.")),
+                >= 0 when size != staged.Length => Observable.Throw<int>(new InvalidOperationException(
+                    $"Staged content '{staged.Handle}' for '{staged.Path}' is {size:N0} bytes but "
+                    + $"the handle declares {staged.Length:N0} — refusing to write a truncated asset.")),
+                _ => target.GetContent(blob)
+                    .SelectMany(stream => stream is null
+                        ? Observable.Throw<int>(new InvalidOperationException(
+                            $"Staged content '{staged.Handle}' for '{staged.Path}' disappeared from "
+                            + $"'{blob}' between the probe and the read, so the file was NOT written."))
+                        : target.SaveFile(dir, name, () => stream).Select(_ => 1))
+            });


Correct, and the contract was the part that was wrong rather than the branch. Refusing on -1 would reject content that is almost certainly intact on every non-seekable store, for ever — so the write still proceeds, but it is no longer silent: CopyStaged logs a warning naming the handle, the path and the declared length when a write goes out unverified, and both the StagedContentFile.Length xmldoc and Doc/Architecture/OutOfBandContentTransfer now state the limit instead of implying a guarantee. Fixed in f946f07. (Every file-system-backed collection reports a size, which is every store a content sync writes to today — the two failing shapes, a missing blob and a genuine mismatch, still throw by name.)

rbuergi · 2026-09-04T07:38:34Z

+    internal static IObservable<Unit> Discard(ContentCollection destination, IEnumerable<string> handles)
+    {
+        var blobs = handles.Distinct(StringComparer.OrdinalIgnoreCase)
+            .Select(h => ContentStaging.BlobPath(h))
+            .ToArray();
+        return blobs.Length == 0
+            ? Observable.Return(Unit.Default)
+            : blobs
+                .Select(blob => destination.DeleteFile(blob)
+                    .Catch<Unit, Exception>(_ => Observable.Return(Unit.Default)))
+                .Concat()
+                .DefaultIfEmpty(Unit.Default)
+                .LastAsync();
+    }


Agreed — that was a silent swallow, and it was fixed in 49e8f7d (pushed before this review landed). Discard and SweepStale both take the builder's logger now: a blob that will not delete is logged as a warning naming the blob and the collection, a reclaimed stale blob logs at Information, and a staging folder that cannot be enumerated logs at Debug. Continuing past a failed delete is still deliberate — one stubborn blob must not strand the rest or turn a successful sync into a failure — but what it leaves behind is now visible rather than inferred.

github-actions · 2026-09-04T07:26:54Z