fix(import): a content sync is many bounded deliveries, never one whole tree (#2885)

src/MeshWeaver.ContentCollections/ContentImportExtensions.cs

@@ -1,3 +1,4 @@
+using System.Collections.Immutable;
 using System.Reactive.Linq;
 using MeshWeaver.Mesh;
 using MeshWeaver.Mesh.Security;
@@ -201,8 +202,15 @@ string FullPath(string rel)
                 if (!request.Mirror)
                     return writes;
 
-                var keep = request.Files
-                    .Select(f => FullPath(f.Path))
+                // 🚨 issue #2885: the WRITES are chunked so no delivery ever carries a whole asset
+                // tree, but the PRUNE is not chunkable — it asks "what is under this folder that the
+                // source no longer carries", and a chunk answering that from its own slice would
+                // delete the other chunks' files. So a split write names its FULL keep set here, in
+                // paths, and the prune stays one authoritative pass. Absent (the unsplit case) the
+                // set is this request's own Files, exactly as before.
+                var keep = (request.MirrorKeepPaths is { } declaredKeep
+                        ? declaredKeep.Select(NormalizePath)
+                        : request.Files.Select(f => FullPath(f.Path)))
                     .ToHashSet(StringComparer.OrdinalIgnoreCase);
 
                 // 🚨 issue #435: when the caller declares which paths the SOURCE owns (a GitSync mirror
@@ -509,7 +517,86 @@ public SyncContentFilesBuilder SourceOwned(IReadOnlyList<string>? sourceOwnedPat
         return this;
     }
 
-    /// <summary>Post the sync to the target node's hub. Cold — subscribe to run.</summary>
+    /// <summary>
+    /// 🚨 <b>THE BUDGET — issue #2885.</b> How many packaged (base64) bytes of file content one
+    /// delivery may accumulate before the next file starts a new one.
+    ///
+    /// <para>It is <c>DeliveryPayloadBounds.MemoryStreamBlockBytes</c>, the tighter of the two
+    /// transport ceilings the mesh declares and the one a failure report about a delivery must
+    /// itself survive — not a number chosen to make a symptom stop. Measured on the base64 form
+    /// because that is what the message actually weighs: <c>System.Text.Json</c> renders a
+    /// <c>byte[]</c> as base64 (×4/3), the packaged payload is then held as a UTF-16 string (×2 in
+    /// bytes), and <c>Utf8JsonWriter.TranscodeAndWriteRawValue</c> rents up to 3 bytes per char to
+    /// write it — so the transient peak is several times the delivery, once per hop.</para>
+    ///
+    /// <para><b>What this bounds and what it does not.</b> It bounds the ACCUMULATION: a delivery
+    /// is never larger than this plus the one file that could not be split, so the delivery's size
+    /// stops being a function of how much content the Space holds. A single file larger than the
+    /// budget still travels whole — that is a different axis (a file that large belongs behind a
+    /// content-store handle, not inline) and it is not the defect this closes.</para>
+    /// </summary>
+    internal const int PayloadBudgetBytes = DeliveryPayloadBounds.MemoryStreamBlockBytes;
+
+    /// <summary>
+    /// What one file costs the packaged payload: its bytes as base64, plus its path. Exact enough
+    /// to partition against, and it never touches the bytes.
+    /// </summary>
+    private static long PackagedCost(InlineContentFile file)
+        => 4L * ((file.Content.Length + 2) / 3) + file.Path.Length;
+
+    /// <summary>
+    /// Splits the accumulated files into deliveries none of which exceeds
+    /// <see cref="PayloadBudgetBytes"/>, except where one file alone does. A file is never split —
+    /// it is the atom the receiving handler writes — so the guarantee is
+    /// <c>delivery ≤ budget + largest single file</c>, which is bounded by the CONTENT rather than
+    /// by how much of it there is.
+    /// </summary>
+    private ImmutableList<ImmutableList<InlineContentFile>> SplitIntoDeliveries()
+    {
+        // An empty sync is ONE request, deliberately: with Mirror it is the "the source carries
+        // nothing here any more" pass, and collapsing it to zero requests would silently skip a
+        // prune the caller asked for.
+        if (_files.Count <= 1)
+            return [_files.ToImmutableList()];
+
+        var deliveries = ImmutableList.CreateBuilder<ImmutableList<InlineContentFile>>();
+        var current = ImmutableList.CreateBuilder<InlineContentFile>();
+        var accumulated = 0L;
+        foreach (var file in _files)
+        {
+            var cost = PackagedCost(file);
+            if (current.Count > 0 && accumulated + cost > PayloadBudgetBytes)
+            {
+                deliveries.Add(current.ToImmutable());
+                current.Clear();
+                accumulated = 0L;
+            }
+            current.Add(file);
+            accumulated += cost;
+        }
+        deliveries.Add(current.ToImmutable());
+        return deliveries.ToImmutable();
+    }
+
+    /// <summary>
+    /// Post the sync to the target node's hub. Cold — subscribe to run.
+    ///
+    /// <para>🚨 <b>Issue #2885 — the write is split, and the PRUNE GOES FIRST.</b> A Space's
+    /// <c>content/**</c> binaries travel inline, so one request per Space made the delivery as
+    /// large as the Space's asset tree (28,484,421 bytes for <c>AgenticBusiness</c>; 106,070,300
+    /// for <c>AgenticEngineering</c>, whose base64 form does not even fit Orleans'
+    /// <c>MaxMessageBodySize</c>). Serialising that threw <c>OutOfMemoryException</c> on a
+    /// production pod — twice, on a payload UNDER every transport bound, because the transcode
+    /// peaks at several times the payload. No bound can fix an allocation that IS the failure, so
+    /// the producer stops making it.</para>
+    ///
+    /// <para><b>Why the mirror rides the FIRST delivery, not the last.</b> The prune is one
+    /// authoritative pass measured against the whole set (<c>MirrorKeepPaths</c>), and putting it
+    /// first makes the split safe under any receiver: were the keep set ever not honoured, the
+    /// files it would wrongly prune are precisely the ones the FOLLOWING deliveries are about to
+    /// write, so the operation still converges on the correct folder. Last-position pruning has no
+    /// such property.</para>
+    /// </summary>
     public IObservable<ImportContentResponse> Post()
     {
         // 🚨 A declared identity (WithAccessContext / ImpersonateAsSystem) wins outright: it is a
@@ -520,19 +607,59 @@ public IObservable<ImportContentResponse> Post()
         // first stage, inside the System scope) and had all 409 SyncContentFilesRequest posts
         // (built in its second stage, outside) failed closed for a null AccessContext.
         var captured = _identity ?? ContentImportExtensions.CaptureCallerContext(_hub);
-        var request = new SyncContentFilesRequest(_targetCollection, _targetPath, _files.ToArray())
-        {
-            Mirror = _mirror,
-            SourceOwnedPaths = _sourceOwnedPaths,
-        };
         var address = new Address(_nodePath);
+        var deliveries = SplitIntoDeliveries();
+        var split = deliveries.Count > 1;
+        // Collection-relative, the form the mirror compares against — and the form
+        // SourceOwnedPaths already uses. Only computed when the write was actually split.
+        var keepPaths = split && _mirror
+            ? _files.Select(f => CombineTargetPath(f.Path)).ToImmutableList()
+            : null;
+
+        var requests = deliveries.Select((files, index) =>
+        {
+            // The mirror is ONE pass and it is the FIRST — see the remark on this method.
+            var prunes = _mirror && index == 0;
+            return new SyncContentFilesRequest(_targetCollection, _targetPath, files)
+            {
+                Mirror = prunes,
+                // An additive chunk prunes nothing, so it carries neither prune input.
+                SourceOwnedPaths = prunes ? _sourceOwnedPaths : null,
+                MirrorKeepPaths = prunes && split ? keepPaths : null,
+            };
+        });
+
         // Off-router issuing, same reason as ContentImportBuilder.Post: the router must be neither
         // end of the request/response pair (ROUTER_TRAFFIC); a non-router hub gets itself back.
-        return ContentImportExtensions.CarryPostIdentity(
-            Observable.Defer(() => _hub.NodeOperationIssuingHub()
+        // Concat, never Merge: the deliveries are sequential so the pod never holds more than one
+        // of them, which is the entire point — and it keeps the prune pass ordered ahead of the
+        // writes that follow it.
+        var posted = requests
+            .Select(request => Observable.Defer(() => _hub.NodeOperationIssuingHub()
                 .Observe(request, o => ContentImportExtensions.ConfigurePost(o, address, captured))
                 .Select(d => d.Message)
-                .Take(1)),
-            _hub, _identity);
+                .Take(1)))
+            .Concat()
+            // Fold the per-delivery answers into the one the caller has always seen. A failure is
+            // reported as itself and stops the sequence: TakeUntil disposes the Concat, so the
+            // deliveries behind it are never posted.
+            .Scan(ImportContentResponse.Ok(0), (total, answer) => answer.Success
+                ? ImportContentResponse.Ok(total.FilesImported + answer.FilesImported)
+                : answer)
+            .TakeUntil(answer => !answer.Success)
+            .LastAsync();
+
+        return ContentImportExtensions.CarryPostIdentity(posted, _hub, _identity);
+    }
+
+    /// <summary>
+    /// A file's path joined onto <see cref="To(string,string)"/>'s folder — the collection-relative
+    /// form the handler's mirror compares against (forward slashes, no leading slash).
+    /// </summary>
+    private string CombineTargetPath(string filePath)
+    {
+        var baseDir = (_targetPath ?? string.Empty).Replace('\\', '/').Trim('/');
+        var rel = (filePath ?? string.Empty).Replace('\\', '/').TrimStart('/');
+        return baseDir.Length == 0 ? rel : $"{baseDir}/{rel}";
     }
 }

src/MeshWeaver.Documentation/Data/Architecture/OversizedDeliveryRefusal.md

@@ -349,11 +349,77 @@ placement of any bound changes that. Note that bounding *concurrency* (as the st
 already does, `BatchSize = 5`) is **not** that axis and does not help: it limits how many deliveries
 are in flight, never how big one of them is.
 
-> **This is why #3046 stays open** while #3047, #3048, #3049 and the rest of that burst do not. Its
-> leg — `BuildPodHubRoute`'s `IPodHubGrain.Deliver` to a bulk-import hub — has been guarded since
-> #2897 and the delivery went through anyway, which means the payload was *under* the bound and the
-> allocation multiple is what killed the pod. Everything above narrows that band; only the producer
-> closes it.
+## The producer that was building it whole (#2885)
+
+The producer axis stayed open for a day because none of the stacks names a message type, only a
+target: `import/{meshHubId}`, the dedicated bulk-import hub. Four incidents converged on it —
+#2885 (twice: 2026-08-31 05:52Z and 2026-09-02 03:08Z) and #3046/#3047/#3048 (2026-09-02
+02:59–03:00Z) — and the identification is arithmetic, not inference.
+
+**The trigger.** `AgenticBusiness/_GitSync` was written at 2026-09-02 02:49:58Z; the burst begins
+five minutes later. That is a forced GitSync re-import, which is exactly the event the first
+measurement on #2885 said a six-hour quiet window could not rule out.
+
+**The message.** A GitSync import mirrors a Space's git-committed `content/**` binaries into its
+content collection with `SyncContentFilesRequest`, whose `Files` carry the raw bytes **inline** —
+deliberately, so binary assets are never round-tripped through a text API.
+`ContentAssetMapper.ToContentSyncs` emits **one** `StaticContentSync` per Space (also deliberately:
+the mirror is meant to be one authoritative pass over the whole collection), and
+`SyncContentFilesBuilder.Post` turned that one group into **one message**. So the delivery's size
+was the size of the Space's entire asset tree.
+
+**The arithmetic.** `Systemorph/MeshWeaver.Education`, the repo `AgenticBusiness` syncs from:
+
+| Space | `content/**` bytes | as base64 | as a UTF-16 `RawJson` string | transcode rent (≤3 B/char) |
+|---|---|---|---|---|
+| `AgenticBusiness` | 28,484,421 (10 files) | ~38 MB | ~76 MB | **~114 MB** |
+| `AgenticEngineering` | 106,070,300 (12 videos) | **~141 MB** | ~283 MB | ~424 MB |
+
+The `AgenticBusiness` payload is comfortably **under** `MaxMessageBodySize`, which is why every
+bound on this page let it through and #3046 could correctly observe that "the leg is guarded and the
+delivery went through anyway". The rent is what failed. `AgenticEngineering` is the other end of the
+same defect and is not an OOM at all: its base64 form exceeds the 100 MiB frame limit outright, so
+once the producer-side refusals landed that Space's assets **could not sync at all** — a silent,
+total failure of a user-visible feature, produced by the same "one delivery per Space" shape.
+
+**The fix — chunk the writes, keep the prune one pass.** `SyncContentFilesBuilder.Post` now
+partitions the files into deliveries against a budget of
+`DeliveryPayloadBounds.MemoryStreamBlockBytes` — the *tighter* of the two transport ceilings this
+page already names, measured on the base64 form because that is what the message weighs — and posts
+them with `Concat`, so the pod holds one at a time. A file is never split: it is the atom the
+receiving handler writes, so the guarantee is `delivery ≤ budget + largest single file`, i.e. the
+delivery's size stops being a function of *how much* content the Space holds.
+
+The prune is the part that does not chunk. It asks "what is under this folder that the source no
+longer carries", and a chunk answering that from its own slice would delete the other chunks' files.
+So `SyncContentFilesRequest` gains `MirrorKeepPaths`: the full keep set as **paths**, which is all
+the prune ever needed and which costs nothing. Exactly one delivery mirrors — **the first**, not the
+last. That ordering is the safety property: were the keep set ever not honoured by the receiver, the
+files it would wrongly prune are precisely the ones the *following* deliveries are about to write,
+so the operation still converges on the correct folder. Last-position pruning has no such property.
+
+An unsplit sync is byte-for-byte what it was: one request, `MirrorKeepPaths` null, the mirror
+measured against its own `Files`.
+
+### What this still does not cover
+
+- **A single file larger than the budget travels whole.** The largest asset in the Education repo is
+  a 9.9 MB video → ~13 MB base64 → ~40 MB rent, which is survivable but is not *bounded* by
+  anything. A file that large belongs behind a content-store handle rather than inline. That is a
+  different defect from "the producer built the tree whole" and is deliberately not conflated with it.
+- **`DeliveryFailure`'s strip does not fire on a typed payload.** `DeliveryPayloadBounds.IsOversized`
+  returns false for anything that is not `RawJson` — by design, since guessing at the size of a CLR
+  object would mean serialising it on the hot path. But a NACK is very often built on the
+  **receiving** hub, where the message has already been deserialised into its CLR type, so the
+  construction invariant introduced in #3044 silently does not apply there and the body is echoed
+  back to the sender verbatim. On the content-sync path the producer fix makes that echo at most one
+  chunk; as a general property of the invariant it remains, and it is worth knowing before relying on
+  the strip.
+- **Other unbounded inbound bodies at `import/{id}`** are smaller but real: `CreateNodesResponse`
+  echoes up to 25 full `MeshNode`s the caller reads only `Path` off; `CreateOrUpdateNodeResponse`
+  echoes the node; `DeleteNodeResponse` carries every deleted path; the import/content manifests are
+  one entry per node and per content file. All are text and grow with the partition rather than with
+  its binaries.
 
 ## Rules
 
@@ -380,6 +446,18 @@ are in flight, never how big one of them is.
 - **A `[PreventLogging]`-style attribute cannot reach a type with its own converter.** The resolver
   strips *properties*; a custom-converted type has none. Check that a redaction attribute is actually
   read before trusting it.
+- **A stack that names only a target address is still identifiable — by arithmetic.** Ask what the
+  producer at that address builds, measure the SOURCE (the git tree, the collection, the partition),
+  and multiply through base64 → UTF-16 → transcode rent. #2885 sat open across four incidents for
+  want of a payload size that no stack carried and that a `du` of the source repository supplied in
+  one command.
+- **When a write is split, the PRUNE is not.** Chunking an operation that also deletes means the
+  delete must be handed the whole set — as identifiers, never as content — and must run as one pass.
+  Put that pass FIRST: any over-prune is then repaired by the writes queued behind it.
+- **A refusal that lands on an unfixed producer converts an OOM into a silent total failure.** Once
+  `AgenticEngineering`'s 141 MB sync exceeded the frame limit, the bound did its job and the assets
+  simply never synced. A bound is a diagnosis, not a remedy: check what the refused producer does
+  next.
 
 ## See also
 

src/MeshWeaver.Documentation/Data/Architecture/StaticRepoImport.md

@@ -208,6 +208,12 @@ Two things make that work end to end, and both are easy to break again:
 
 Test: `PackageContentAssetInstallTest` — the bytes land at the exact layout `/api/content/{root}/{file}` resolves to, a text asset still round-trips, and an untracked upload survives a re-install.
 
+### 🚨 One sync is MANY deliveries — the bytes are never sent whole (#2885)
+
+`ContentAssetMapper.ToContentSyncs` still emits **one `StaticContentSync` per Space** — the mirror has to be one authoritative pass over the whole collection — but `SyncContentFilesBuilder.Post` no longer turns that into one message. It partitions the files against `DeliveryPayloadBounds.MemoryStreamBlockBytes` (measured on the base64 form, which is what the message weighs) and posts them with `Concat`, so the portal ever holds one batch. A whole-Space message was 28 MB of course video for `AgenticBusiness` — ~114 MB of transient allocation per hop, which took a production pod down twice — and 141 MB packaged for `AgenticEngineering`, over the frame limit entirely, so that Space's assets silently stopped syncing once the producer-side refusals landed.
+
+The prune does not chunk: it rides the **first** delivery and carries `SyncContentFilesRequest.MirrorKeepPaths`, the full keep set as paths. First rather than last is the safety property — an over-prune would be repaired by the writes queued behind it. Full argument and arithmetic: [Oversized Delivery Refusal](/Doc/Architecture/OversizedDeliveryRefusal) → "The producer that was building it whole". Tests: `ContentSyncIsNeverBuiltWholeTest`, `ContentSyncMirrorSurvivesTheSplitTest`.
+
 ## The two primitives
 
 ### 1. Source fingerprint — the content-version