docs(architecture): the end vision — jobs, durable streams, document ingestion, complex flows, and THE plan by rbuergi · Pull Request #6017 · Systemorph/MeshWeaver

rbuergi · 2026-10-03T11:22:53Z

Why

Maintainer, 2026-10-03: document the end vision ("the full monty") as the authoritative design, with a precise gap analysis — "please factor into docs and plan". Platform-wide designs go to core (Doc/Architecture); the module pages link here.

What

Doc/Architecture/DocumentIngestionEndVision — one page:

Decisions: every activity a job in a queue on control (queue/job NodeTypes, the queue's hub IS the queue, express > babysitter > the rest, capability routing to pre-installed instances >= requirements, the fluent declaration with retries default 5); durable streams as saved nodes with checkpoints (lease = live subscription, orphan adoption exactly once, park = waiting for the unblocking event); logs and documents as chunked nodes in the owning partition + the Storage settings section; the correlation map over synced streams; triage that acts; the bug-fix process; self-diagnosis; the repo split (insurance → MeshWeaver.Reinsurance).
The ingestion: attach → content table or PARK on a governed activity with a user-named approver → idempotent InitializeRequest → one job per distinct content on the builds queue, jobs spawning jobs → extract (PdfPig, DOCX, e-mail, local OCR) → label (taxonomy roles, addresses, message types) → chunk + index; the state machine; how exactly-once is guaranteed (content-hash identity, atomic insert-if-absent claim, the document's own hub owns "parsed", durable checkpoints); local models; the default taxonomy.
Complex flows: ask-and-await, retry-then-flag, triage-unblock, correlation.
The executable specs step by step (MeshWeaver.Plugins fix(io): require IoPoolRegistry on file-system adapters — no ledgerless Unbounded fallback (#613 follow-up a) #2736 Testing/EndVision + Testing/ComplexFlows; insurance scenario MeshWeaver.Reinsurance fix(chat): full-screen home composer; CLI-harness command routing; honest effort #245) and how they run (a job routed to a pre-installed test instance, no builds).
THE plan: 27 gaps in dependency order — repo · module, size, prerequisites, status · owner (chore: retire the #2721 type-forwards.allow entries (stale the moment #2721 merged) #2730, feat(mesh): document parts get their own satellite tables + the document-log contract #6013, Module MVID skew: a publication's dependency records name one build of AI/Collaboration, the registry serves another under the same version — every consumer gate declines (root cause of the bake-shortfall saga) #2698, or nobody), and the e2e assertion that proves each — plus what is already landed and what the code says today (with the evidence).

Related links added from DurableStreamsViaMeshNodes, GovernedAutonomy and OnboardingNewEnvironment (the new-environment guide).

Docs only; no code. Links point to pages on main (DocumentParts is referenced by name until #6013 lands).

🤖 Generated with Claude Code

…ingestion, complex flows, and THE plan One authoritative page for the architecture decided on 2026-10-03: every activity a job in a queue on control (queues and jobs as NodeTypes, the queue's hub IS the queue, express > babysitter > the rest, capability routing to pre-installed instances, one fluent declaration with retries default 5), durable streams as saved nodes with checkpoints (lease = live subscription, park = waiting for the unblocking event), logs and documents as chunked nodes in the owning partition, the correlation map, triage that acts, the bug-fix process, self-diagnosis and the repo split. The document-ingestion end vision step by step (attach → content table or governed park → idempotent InitializeRequest → one job per content, exactly once → extract, label, discover, chunk, index), its state machine, how the mesh guarantees exactly-once, local models, the taxonomy; the four complex flows (ask-and-await, retry-then-flag, triage-unblock, correlation); the two executable specs (MeshWeaver.Plugins #2736, insurance scenario MeshWeaver.Reinsurance #245) step by step; and THE plan — 27 gaps in dependency order with repo, size, owner/PR, status and the assertion that proves each. Related links from DurableStreamsViaMeshNodes, GovernedAutonomy and OnboardingNewEnvironment. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

github-actions · 2026-10-03T11:28:08Z