Skip to content

A Space whose content sync is refused stops syncing SILENTLY — AgenticEngineering's assets are over the frame limit outright #3101

Closed

Description

@rbuergi

A Space whose assets stopped syncing, silently

Found while fixing #2885 (the import-hub OOM), and not the same defect — #2885's producer fix does not close this one, so it needs its own tracking.

MeshWeaver.Education's AgenticEngineering Space holds 106,070,300 bytes under content/**. A GitSync content sync carries those binaries inline, and base64 inflates them:

Space content/** base64 form verdict
AgenticBusiness 28,484,421 B ~38 MB under MaxMessageBodySize — this was #2885's OOM
AgenticEngineering 106,070,300 B ~141 MB over the frame limit outright

So once the oversized-delivery refusals landed (#1890 / #2897 / #3018 / #3032), that Space's content sync began being refused rather than delivered. The assets stopped syncing — and nothing said so.

Why this is worse than an OOM

An OOM is loud: it kills a pod, files an incident, and someone looks. A refusal is quiet by design — the refusal is the correct behaviour of the guards, working exactly as intended. There is no error surfaced to whoever authored the content, no ledger entry saying "this Space's assets are stale", and nothing that distinguishes "synced and unchanged" from "refused every time since the guards landed".

The person who notices is a learner opening a course page whose video or image is missing.

Relationship to #2885 / #3097

#3097 partitions the sync so a delivery is ≤ budget + largest single file, which fixes AgenticBusiness and every Space whose aggregate was the problem. It does not fix this one on its own if any single file in AgenticEngineering is itself over budget — #3097's own residual, recorded in Doc/Architecture/OversizedDeliveryRefusal:

a single file over budget still travels whole — a different axis, belonging behind a content-store handle.

So the two questions this issue should answer, in order:

  1. Does AgenticEngineering now sync after fix(import): a content sync is many bounded deliveries, never one whole tree (#2885) #3097? If its files are individually small and only the aggregate was over, fix(import): a content sync is many bounded deliveries, never one whole tree (#2885) #3097 already fixed it and this issue closes with a measurement. That is the cheap check and it should be done first.
  2. If not, which files are individually over budget? Those need the content-store handle rather than an inline payload — a real design change, not a bound.

The part that should be fixed regardless

Whatever the answer to (1), a Space whose assets are being refused should say so. A refusal that is correct at the transport layer is still a delivery failure at the content layer, and today it is indistinguishable from success. Some durable signal — a ledger entry, a sync status on the Space, a surfaced verdict — is what turns this from "a learner reports a missing video" into "the sync reports its own state".

Provenance

Measured by the #2885 investigation against Systemorph/MeshWeaver.Education. The byte counts above are from that repo's working tree; re-measure before acting, since content moves.

Activity

  1. rbuergi commented on Sep 2, 2026

    @rbuergi
    ContributorAuthor

    Partial measurement, and an honest note about why it is partial

    The issue asks for the cheap check first — "Does AgenticEngineering now sync after #3097?" — so I started there rather than with the design half.

    What I could read (memex mesh, via MCP, 2026-09-02 ~19:00Z):

    get @AgenticEngineering/content/        →  [ {videos, isFolder: true}, {og.png, 2026-08-27} ]
    get @AgenticEngineering/content/videos/ →  Error: File 'videos' not found in collection 'content'
    

    So the collection lists a videos folder and then cannot enumerate it. That is suggestive of the shape this issue describes — a folder entry with no files behind it — but it is not conclusive: it may equally be a listing-API quirk for nested paths, and I have not separated the two.

    What I could not read. The follow-up (get @AgenticEngineering, to pull the video paths out of the page body and probe them individually) came back:

    Unavailable: AgenticEngineering — this read reached no verdict, so it is UNKNOWN whether
    this node exists. This is NOT 'not found' … Cause: read reached no verdict within 10s
    

    That is the tri-state behaving correctly, and I am taking it at its word rather than treating it as evidence: a non-verdict is not a fact. The deployment is under a manual roll right now and #3146's import storm is live on the same namespace, so this is the wrong moment to conclude anything about whether a Space's assets are present.

    So question 1 is still open, and it needs re-running on a quiet portal. I would rather say that than record a measurement I do not trust — the whole subject of this issue is a silent state being mistaken for a healthy one.

    The part that does not depend on the answer

    The issue's own framing already settles what matters most, and I want to underline it because it is independent of whether AgenticEngineering currently syncs:

    a Space whose assets are being refused should say so.

    Right now "synced and unchanged" and "refused on every attempt since the guards landed" are indistinguishable from outside. The refusal is correct behaviour at the transport layer — that is what makes it invisible: nothing is failing, so nothing reports. The person who finds out is a learner opening a page with a missing video.

    That is the same defect class as three others fixed today, all of which were "the component behaved correctly and the state it left was unobservable":

    A durable signal on the Space — a ledger entry, a sync status, a surfaced verdict — is worth more than the answer to question 1, because it converts every future occurrence from "a learner reports a missing video" into something the sync says about itself. It is also a real feature rather than a repair, and wants its own change.

    Disposition

    No code changed. Question 1 needs a re-measurement on a quiet deployment; question 2 depends on it; the durable-signal half stands on its own and is the piece I would build first regardless of how the first two land.

    Measured 2026-09-02 against the memex mesh. Re-measure before acting — content moves, and this reading was taken mid-roll.

  2. rbuergi commented on Sep 2, 2026

    @rbuergi
    ContributorAuthor

    Part 3 has landed — #3149, merged 2026-09-02

    The part that should be fixed regardless: whatever the answer to (1), a Space whose assets are being refused should say so.

    That is now true. A content sync that does not succeed is no longer folded into 0 files:

    • both content passes carry a named set of refused owning nodes instead of a bare count;
    • the previously silent Success == false arm logs, naming the node and stating its assets are not in the mesh;
    • the refusal reaches the import's Outcome, which becomes ImportedWithRefusedContent → Warning, and the summary names the refused nodes.

    🚨 The last point is the one that matters most, and it is not the one that looks most important. The attempt activity is the human-readable log, but the lock is stamped from the Outcome, and the lock is the durable short-circuit. My first version escalated only the attempt's status — it would have read correctly and changed nothing, leaving the Space skipped for ever under a fix everyone believed had landed. The control arm removes exactly that one arm and prints outcome = Imported, so the test measures the load-bearing edit rather than the cosmetic one.

    Practical effect: a Space whose assets were refused is stamped Warning, so the next boot re-attempts instead of short-circuiting on a green marker. When the transport problem is fixed, the assets arrive on the next sync with nobody remembering to force one.

    🚨 Deliberately NOT closing — parts 1 and 2 are untouched

    1. Does AgenticEngineering sync now, after fix(import): a content sync is many bounded deliveries, never one whole tree (#2885) #3097? Still unanswered. I tried and stopped: the mesh returned "this read reached no verdict … within 10s" while the deployment was mid-roll, and treating a non-verdict as evidence is precisely the mistake this issue is about. content/ lists a videos folder whose enumeration then fails, which is suggestive and not conclusive — it may equally be a listing quirk for nested paths, and I did not separate the two.
    2. If not, which files are individually over budget? Depends on (1).

    This issue's own residual stands too, quoted from Doc/Architecture/OversizedDeliveryRefusal: a single file over budget still travels whole. #3097 partitions by aggregate; AgenticEngineering's ~106 MB (~141 MB base64) is over the frame limit outright, so if any individual file exceeds the budget it needs a content-store handle — a design change, not a bound.

    What has changed is that the difference is now observable. Re-running question 1 on a quiet portal is the next step, and the answer is no longer something you have to infer from a learner reporting a missing video: the import says it.

    #3149 merged 2026-09-02. Re-measure question 1 before acting — content moves, and the earlier reading was taken mid-roll.

  3. rbuergi commented on Sep 3, 2026

    @rbuergi
    ContributorAuthor

    Questions 1 and 2, measured — and one of them had a wrong control

    🚨 The content/videos/ enumeration error is a listing quirk, not evidence

    The earlier reading stopped at exactly the right place — "it may equally be a listing quirk for
    nested paths, and I did not separate the two"
    . Separated now, with a control:

    get @AgenticEngineering/content/  → [ {videos, isFolder:true}, {og.png, 2026-08-27} ]
    get @AgenticEngineering/content/videos/ → Error: File 'videos' not found in collection 'content'
    
    get @AgenticBusiness/content/     → [ {videos, isFolder:true}, {og.png, 2026-09-02} ]
    get @AgenticBusiness/content/videos/    → Error: File 'videos' not found in collection 'content'
    

    AgenticBusiness is the Space that is under the limit — #2885's OOM case, i.e. the one known to
    be deliverable — and it fails identically. So that error says nothing about whether assets are
    present. It is a quirk of enumerating a nested collection folder. Anything reasoning from it,
    including the "suggestive" note in the earlier comment, should be discarded.

    🚨 Question 2: NO single file is over budget — only the aggregate is

    Measured against main of Systemorph/MeshWeaver.Education, AgenticEngineering/content/videos:

    bytes base64 vs MaxMessageBodySize (104,857,600)
    aggregate (24 files) 105,728,384 ~141 MB over
    largest single file (module1-intro.mp4) 9,891,614 ~13.2 MB comfortably under

    Every one of the 12 .mp4s is 7.7–9.9 MB; every poster is ~0.33–0.37 MB.

    This retires the residual this issue was holding open. The quoted worry was:

    a single file over budget still travels whole … if any individual file exceeds the budget it needs
    a content-store handle — a design change, not a bound.

    No individual file exceeds the budget, so that design change is not required for this Space.
    Aggregate partitioning (#3097) is the right and sufficient mechanism here. The residual remains true
    as a general statement about the transport, but it is not what is stopping AgenticEngineering, and
    it should not gate this issue.

    Question 1: the sync has not ATTEMPTED since the assets came back

    The premise moved under the issue, which is worth stating plainly:

    • Systemorph/MeshWeaver.Education#235 — "restore the 12 course videos and posters to git" —
      merged 2026-08-30. Those 24 binaries had been deleted by 07c4d90a on 2026-07-18 and were
      restored by that PR. So the ~106 MB is only back in git as of 08-30.
    • Both Spaces' sync config reads lastSyncedAt: 2026-08-27T13:58 (AgenticEngineering at
      61cbbac, AgenticBusiness at 550d5b7) — three days before the restore.

    So AgenticEngineering has not attempted a content sync since its assets existed in git, and the
    refusal hypothesis is untested post-#235 rather than confirmed. Note also that
    memex.meshweaver.cloud serves these files from its own store — #235 records they were uploaded there
    by hand on 07-17/18 — which is why content/ lists a videos folder at all and why this portal is
    the wrong instrument for the question.

    What this leaves

    1. Trigger a sync of AgenticEngineering on a repo-backed install (not memex, whose copy is
      hand-uploaded) and read the import outcome — fix(import): a refused content sync is not a successful import (#3101) #3149 made a refusal observable there, so the answer
      is now readable rather than inferred.
    2. If it is refused, the fix is aggregate partitioning coverage, not a content-store handle.

    Measured 2026-09-03. Sizes from the GitHub contents API at main; budget from
    MessageSizeGuard (104,857,600).

  4. rbuergi commented on Sep 3, 2026

    @rbuergi
    ContributorAuthor

    Q1 answered: no — #3097 does not fix AgenticEngineering. And the scope is wider than this issue assumed.

    Re-measured on MeshWeaver.Education@61cbbac (the issue asks for exactly this before acting).

    The budget is 1 MB. ContentImportExtensions.PayloadBudgetBytes = DeliveryPayloadBounds.MemoryStreamBlockBytes = 1 << 20 — Orleans' memory-stream block size, "the tighter of the two transport ceilings on one message", hard-coded in MemoryAdapterFactory with no configuration surface.

    #3097's guarantee is delivery ≤ budget + largest single file, and a file is never split. So the question is not the aggregate at all — it is whether any single file's packaged cost exceeds 1 MB. Using the partitioner's own cost function (4 * ((len + 2) / 3) + path.Length):

    Space files over budget total largest packaged largest file
    AgenticEngineering 25 12 101.2 MB 12.6 MB content/videos/module1-intro.mp4
    AgenticBusiness 9 4 27.2 MB 10.4 MB content/videos/agentic-business.mp4
    AgenticPrimerDe 7 3 10.9 MB 4.1 MB content/videos/primer1de.mp4
    AgenticPrimer 7 3 9.8 MB 3.7 MB content/videos/primer1.mp4
    AdvancedBusinessRules 2 1 9.5 MB 12.0 MB content/videos/advanced-business-rules.mp4
    DataModeling 3 1 8.6 MB 10.4 MB content/videos/datamodeling.mp4
    AgenticOffice 3 1 8.2 MB 9.8 MB content/videos/agentic-office.mp4

    Every Space in the repo has at least one file over budget — 25 files in total. AgenticEngineering's largest packages to 12.6 MB, twelve times the ceiling.

    Two corrections to this issue's framing

    1. AgenticBusiness was not fixed by fix(import): a content sync is many bounded deliveries, never one whole tree (#2885) #3097 either. This issue records it as "under MaxMessageBodySize — this was Routing a message delivery to an import hub exhausts pod memory during JSON serialization in MessageDeliveryConverter #2885's OOM", with fix(import): a content sync is many bounded deliveries, never one whole tree (#2885) #3097 closing it. Its aggregate was the problem for the OOM, but four of its nine files are individually over budget, the largest packaging to 10.4 MB. It is in the same state as AgenticEngineering, one order of magnitude down.

    2. The axis is "has a video", not "is large". AdvancedBusinessRules totals 9.5 MB — the third smallest Space here — and carries the second-largest single file (12.0 MB packaged). Sorting Spaces by total size, which is what this issue did, does not identify the affected set. The threshold is a raw file over ~768 KB, which every video in the repo clears.

    These counts are a lower bound: the effective ceiling is below 1 MB, because the delivery's JSON envelope rides in the same message as the file's base64.

    So Q2 is the live question, for all seven Spaces

    "which files are individually over budget? Those need the content-store handle rather than an inline payload — a real design change, not a bound."

    That is the residual #3097 recorded for itself in Doc/Architecture/OversizedDeliveryRefusal ("a single file over budget still travels whole — a different axis, belonging behind a content-store handle"). It is now the whole of what remains here, and it affects the entire course catalogue rather than one Space.

    The reporting half looks done — worth confirming separately

    #3149 ("a refused content sync is not a successful import") merged on 2026-09-02 against this issue and appears to cover "a Space whose assets are being refused should say so". Someone should confirm a refused sync now produces the durable signal on one of the Spaces above, since that half can be verified independently of the design change.

    Reproducing

    git ls-tree -r -l origin/main | awk '$5 ~ /\/content\// {
      raw=$4+0; b64=4*int((raw+2)/3)+length($5);
      n=split($5,p,"/"); s=p[1]; tot[s]+=raw; cnt[s]++;
      if (b64 > 1048576) over[s]++
    } END { for (s in tot) printf "%-22s files=%d over=%d total=%.1fMB\n", s, cnt[s], over[s]+0, tot[s]/1048576 }'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions