delete-space leaves an empty same-named Space behind: the drain re-provisions the partition it just removed, so a removal cannot be verified by listing #3436

Activity

1. rbuergi commented on Sep 6, 2026

   

   rbuergi

   on Sep 6, 2026

   ContributorAuthor

   More actions

   Root cause established: the drop handler is registered PER NodeType, and only for two of them

   PartitionDropPostDeletionHandler matches on a single NodeType string supplied at its registration
   site:

   public string NodeType => nodeType;   // AddSpaceType → "Space", AddUserType → "User"

   It is constructed in exactly two places in src/:

   • src/MeshWeaver.Graph/Configuration/SpaceNodeType.cs:202
   • src/MeshWeaver.Graph/Configuration/UserNodeType.cs:57

   The four partitions that survived were rooted at Store/Plugin nodes, not Space ones — from
   the run's own preview details:

   delete-space AgenticPrimerDe   → exists: Store/Plugin "Agentic Primer — Wünsch dir Software!" — 59 node(s)
   delete-space DataImportExport  → exists: Store/Plugin "Data Import & Export"                  — 44 node(s)
   delete-space DataModeling      → exists: Store/Plugin "Data Modelling"                        — 429 node(s)
   delete-space ThinkInStreams    → exists: Store/Plugin "Think in Streams"                      — 90 node(s)
   

   Every partition that was correctly and completely removed — the whole Reinsurance family plus
   AdvancedBusinessRules and AgenticEngineering — was rooted at a Space. Confirmed against
   Admin/Partition/* (106 entries): none of the Reinsurance partitions remain, and exactly these four
   do.

   So the handler never ran for a Store/Plugin-rooted partition. The class's own doc comment
   predicts this exact failure for a different type:

   A User home that lacked this handler was left entirely behind (2026-07-19 memex-cloud
   incident); registering it for every OwnsPartition type is the fix.

   Store/Plugin is the third type. The fix stated there was never generalised — it was applied twice,
   by hand, at two registration sites. The registration should be driven by OwnsPartition being
   true, not by a hand-maintained list of two, and a guard should assert that every OwnsPartition
   NodeType has a teardown handler, so a fourth type cannot repeat this.

   This also corrects my earlier hypothesis in the issue body: partition provisioning racing the drain
   is NOT the mechanism. The drop simply never happened, and the empty shell is what a later
   provisioning left behind on a partition that was still fully alive underneath.

   It also explains the asymmetry the issue asked about

   AgenticPrimer (57 nodes) was Store/Plugin too and did NOT come back, while AgenticPrimerDe (59
   nodes) did — so nodeType alone does not predict the SHELL. But it does predict the SCHEMA, which is
   the part that matters: whether a visible shell reappears depends on whether anything subsequently
   touched the partition, while the orphaned schema is there either way. A shell is the visible
   subset of the damage; absence of a shell is not evidence the partition was dropped. Any audit must
   read Admin/Partition/* and the schema list, never the Space listing.

   Third symptom: the orphan is self-protecting

   Attempting the cleanup through the sanctioned API as the maintainer's own identity:

   Delete permission denied for 'AgenticPrimerDe'. The current identity does not hold Delete here — do not retry.
   

   …for all four. The re-provisioned partition carries a fresh default PartitionAccessPolicy
   (AgenticPrimerDe/_Policy, created 67 ms after the root) which does not grant Delete to the identity
   that deleted the original. So the orphan cannot be cleaned up through the same path that created it.

   That makes the severity higher than "leftover shell": a partition can reach a state where it is
   undeletable by ordinary means and invisible in a Space listing.

   Scope of the live damage on the systemorph staff portal

   Four orphaned partitions, all still registered in Admin/Partition:

   partition	definition state
   AgenticPrimerDe	v1, re-created 2026-09-06T10:55:37.968Z
   DataImportExport	v1, re-created 2026-09-06T10:55:39.520Z
   ThinkInStreams	v1, re-created 2026-09-06T10:55:53.405Z
   DataModeling	v4, 2026-06-12 — never deleted at all ("Data Modeling Masterclass")

   DataModeling is the clearest case: its definition predates the removal entirely, so the delete
   never touched it.

   🚨 Do not confuse it with DataModelling (double L, "Data Modelling"), which is a LIVE package and
   partition and must not be touched.

   Worth a separate look, not part of this issue

   The same listing shows several probable historical orphans that no current Space accounts for —
   BinaryClicker, BinaryClickerV2, binary-clicker, binary-clicker-space (four variants of one
   thing), BusinessModels beside BusinessModel, Documentation beside Doc, Providers beside
   Provider, RFPProcessor, UWDeepfield, UWDeepfieldData. These are consistent with the same
   defect operating over months. An audit that reconciles Admin/Partition/* against live partition
   roots would size it; V03_DropRogueSchemas is the precedent for the cleanup shape.
2. rbuergi

   added 2 commits that reference this issue on Sep 6, 2026

   doc(#3436): point the residual-race note at its tracking issue

   ...

   f5caac6

   test(#3436): use TestTimeouts instead of literal 30 s waits

   ...

   cca56e8
3. rbuergi

   closed this as completedin #3449on Sep 6, 2026
4. rbuergi commented on Oct 11, 2026

   

   rbuergi

   on Oct 11, 2026

   ContributorAuthor

   More actions

   (re-posted 2026-10-11; originally by @rbuergi on 2026-09-06)

   Fix in #3449 — and one correction to the proposed generalisation

   The mechanism in the comment above is confirmed. The proposed fix is not sufficient, though, and
   that turned out to be the load-bearing detail.

   Driving the registration off NodeTypeDefinition.OwnsPartition == true would still have missed all
   four victims:

   • Store/Plugin is declared in mesh CONTENT, so no src/-side scan can enumerate it — as the
     issue says. But also:
   • Store/Plugin does not set OwnsPartition at all. Read off the live node
     (get @Store/Plugin, systemorph): its NodeTypeDefinition carries no ownsPartition field, and
     the serializer omits defaults. It does not need one — PackageInstaller.EnsurePartitionsProvisioned
     provisions the target partition, not OwnsPartitionProvisioningValidator. So partition creation
     has two triggers and only one of them consults OwnsPartition; a runtime scan for it would have
     skipped Store/Plugin too.

   So #3449 keys the teardown on the only predicate with no blind spot — the shape of the deleted
   node. A partition is a first path segment, so a top-level node IS its partition's root, whatever its
   type and whenever that type arrived (PartitionDefinition.IsPartitionRoot). Registered once, by
   AddGraph. Details, both guards and the falsification are in the PR and in
   Doc/Architecture/PartitionTeardown.

   The empty shell, explained

   EnsurePartitionBootstrap → ProvisionAndCreateRoot: a child write into a partition whose ROOT row
   is missing provisions the schema and writes a generic Space root named after the path segment.
   That is exactly the observed shape (Store/Plugin → Space, display name → bare id, _Policy
   67 ms later), and it is a second trigger for partition creation behind
   OwnsPartitionProvisioningValidator's back. The writer was almost certainly
   InstalledPackageRepairService, which re-asserts PackageInstaller.EnsureDeclaredAccess for every
   recorded install — and the install record lives in the Plugins partition, so deleting the
   installed partition never removes it.

   That also explains the asymmetry the issue asked about: the shell appears only where something
   subsequently wrote into the partition. #3449 gates the absent-root branch on the partition's backing
   store not being definitively gone, which closes the steady-state loop. It does not close a child
   write that lands while the delete is still draining — see #3451.

   ---

   The orphan audit (read-only; nothing on systemorph was changed)

   Reconciled Admin/Partition/* (106) against live partition roots. 🚨 The root set is not just
   Space — three in-mesh root NodeTypes are in play on this portal:

   Root NodeType	Count	Declared in
   Store/Plugin	67	plugins package content
   Space	16	core src/
   Crm/Client	7	Crm package content
   Store/Catalog	1 (Store)	Store package content

   Every candidate was then confirmed with an exact-path read (get @{partition}), not with the
   query alone: scope:children listings are partition-scoped and RLS-filtered — Admin itself does
   not appear in an unscoped nodeType:Space scope:children even though its root exists — so a query's
   count: 0 is not by itself evidence of absence.

   21 partition definitions have no live root:

   Partition	Definition version / last modified
   BinaryClicker	v4 · 2026-06-30
   BinaryClickerV2	v4 · 2026-07-01
   binary-clicker	v4 · 2026-06-30
   binary-clicker-space	v4 · 2026-07-01
   BusinessModels	v4 · 2026-06-15
   Documentation	(no version recorded)
   RFPProcessor	v4 · 2026-07-14
   UWDeepfield	v4 · 2026-07-09
   UWDeepfieldData	v4 · 2026-07-10
   ClientAlpha	v2 · 2026-08-13
   TestEmail	v2 · 2026-08-12
   EAA	v1 · 2026-08-28
   Helvetia	v1 · 2026-08-19
   MusicIns	v1 · 2026-08-19
   PQTechnology	v1 · 2026-08-27
   SchenkerLabs	v1 · 2026-08-27
   Valucor	v1 · 2026-08-27
   VHVGruppe	v1 · 2026-08-19
   LouisBraun	v1 · 2026-08-20
   RobMind	v1 · 2026-08-26
   StefanBernegger	v1 · 2026-09-01

   Every suspect named in the issue is in that list, and the shape of the residue matches the defect:
   the four binary-clicker* variants, BusinessModels beside the live BusinessModel, Documentation
   beside the live Doc, and three user-home-shaped names (LouisBraun, RobMind, StefanBernegger)
   which fit the pre-2026-07-19 User half of the same bug.

   Also worth a look, the converse: 9 live roots have no partition definition — AI, Crm,
   LinkedIn, Manufacturing, Marketing, SocialMedia, Video, X, YouTube (plus ClientA,
   a live Crm/Client). Harmless in itself (a definition is not required to route), but it means the
   Admin/Partition listing is not a complete index of partitions in either direction.

   And the four from this issue now exist as empty Space shells rather than as missing roots:
   AgenticPrimerDe, DataImportExport, DataModeling, ThinkInStreams.

   What the audit could not do, and what would finish it

   The authoritative third list is the schemas themselves — SELECT nspname FROM pg_namespace ORDER BY 1
   on the portal database. A definition can be missing while the schema is there, and vice versa, so
   until that is read the 21 above are candidates, not a confirmed schema census. That needs a DB
   read I deliberately did not perform.

   No cleanup was run and none is proposed here. V03_DropRogueSchemas (MeshWeaver.Plugins
   src/Memex.Database.Migration/Migrations/) is the precedent for the shape if a migration is
   warranted; the procedure is written up in Doc/Architecture/PartitionTeardown. #3449 only stops NEW
   orphans.