DeleteNode pre-validate-descendants: 150 of 150 Release records of an orphan NodeType never answer ValidateDeleteRequest — the orphan cannot be deleted

What is failing

Deleting a NodeType node whose only descendants are Release/* records is refused because the pre-validation fan-out never gets an answer from any of them:

$ delete Edu/Course        (MCP, memex.systemorph.com, 2026-09-02 ~07:40Z, as rbuergi)
Error deleting Edu/Course: Delete of 'Edu/Course' exceeded 30s timeout in stage 'pre-validate-descendants':
[DeleteNode:pre-validate-descendants] 150 of 150 descendant(s) of 'Edu/Course' did not answer ValidateDeleteRequest
within 30s, so the delete was refused with the subtree untouched: Edu/Course/Release/20260720104632-_-O-9kLO,
Edu/Course/Release/20260720135123-FrXaNpG2, … (+140 more)

Edu/Course is an orphan NodeType (its package retired it on 2026-09-01; compilationStatus: Error, 0 instances — see #2951). Its 150 descendants are all Release/<stamp> nodes written by earlier successful compiles. get @Edu/Course/* returns [] — the release records are not children the listing shows, yet the delete stage enumerates and waits on every one of them.

Why it is a defect, not a slow mesh

150 of 150 silent within 30 s is a 100 % non-answer, not tail latency. Either ValidateDeleteRequest is routed to a per-node hub these Release nodes cannot activate (their type's hub is the broken NodeType itself, or a Release record has no handler for the request), or the fan-out is serialised behind one activation that never completes. Both are deterministic for this shape.
The consequence is that the one repair for a dangling / orphan NodeType — deleting it — is unavailable through the API (MCP delete, and presumably DeleteNodeRequest from any hub). Doc/Architecture/DanglingNodeTypes (A node can reference a NodeType that no longer resolves: two holes, each needing a decision #2993 / Refuse an update that dangles a NodeType; report a prune that strands instances #3042) makes the orphan condition visible; this makes it un-fixable.
The refusal itself is CORRECT behaviour (an unanswered validation must not be read as consent). What is wrong is upstream: descendants that cannot answer, or a fan-out that asks nodes that have nothing to validate.

Where to look

The pre-validate-descendants stage of the delete pipeline (DeleteNode in MeshWeaver.Mesh.Operations / MeshExtensions): how it enumerates descendants (it sees 150 that @Edu/Course/* does not list — the Release container is a satellite-shaped subtree) and to which address it posts ValidateDeleteRequest for a Release record.
Whether Release records need a validation round-trip at all: they are compile artefacts owned by the NodeType, not user content — a deletion of the type should be able to take them with it without asking each one.
Reproduce: any NodeType with N release records whose compile is red (or whose type hub cannot activate), then DeleteNodeRequest on the type.

Acceptance

Deleting an orphan NodeType with release records succeeds (the records go with it) or is refused with a reason that names the real blocker — never a 30 s wall of silence.
A test that stands up a NodeType with release records, breaks its compile, and deletes it.

Found while acting on #2951.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G4xPnGYEbdu8AVvR5jytyj

Root cause: pre-validation is addressed to each descendant's OWN hub, so a NodeType that cannot compile makes its own descendants unanswerable — and deleting it is the one repair that needs them to answer.

MeshExtensions.cs:3222, the pre-validate-descendants fan-out:

var perPath = descendants.Select(p => issuingHub
    .Observe(new ValidateDeleteRequest(p, rootPath), o => callerAccessContext is null
        ? o.WithTarget(new Address(p))
        : o.WithTarget(new Address(p)).WithAccessContext(callerAccessContext))

WithTarget(new Address(p)) routes each request to that descendant's per-node hub. Those hubs are configured from the owning NodeType's configuration lambda. Edu/Course is exactly the type whose configuration does not compile (compilationStatus: Error, #2951), so none of its 150 Release/* hubs can be configured, none activate, and none answers.

That is why it is 150 of 150 and not 149 — it is not tail latency, it is structural. Every descendant of a broken NodeType is unanswerable by construction, and the count will always equal the descendant count.

The catch-22

The only repair for a dangling / orphan NodeType is to delete it. Deleting it requires a ValidateDeleteResponse from every descendant. Every descendant's ability to answer depends on the very NodeType that is broken. The repair requires the broken thing to work.

Why the existing hardening cannot help

#1198 made this failure legible — the message now names the paths and the stage instead of an anonymous timeout, and HandleValidateDeleteRequest deliberately runs on the shorter NestedTimeout rung so a descendant can be the one to give up and say which node and which read. That work is doing its job here: the report is precise. But it assumes the descendant's hub runs. When the hub never activates there is no handler to reach the nested rung at all, so the caller's 30 s bound is the only one that ever fires.

The fix shape (not implemented — deliberately)

The descendant's own hub is consulted so type-specific validators can object. A NodeType whose configuration does not compile contributes no validators — there is nothing to lose by not asking it. So:

When a descendant's owning NodeType is not compilable, validate that descendant on the ISSUING hub — storage read plus the issuing hub's validator chain — instead of messaging a hub that cannot exist.

That is strictly not weaker than today (it runs the same validators minus ones that provably do not exist), it is decidable from the issuing hub without messaging anything (read the NodeType's compilationStatus), and it removes the circular dependency rather than widening a bound.

🚨 Deliberately not proposed: raising the 30 s timeout, or treating a non-answer as "no objection". The first cannot help — the answer is never coming, so a longer wait only fails later. The second would silently skip real validators for every slow-but-healthy descendant, which is a much worse trade than the bug it fixes.

I stopped short of implementing this: it changes a delete path's validation semantics, and it deserves its own change with tests for the compilable and non-compilable branches rather than being folded into a session that was fixing CI. The diagnosis above is what was missing; the change is now a well-specified piece of work.

Operationally, right now

Anyone blocked on removing Edu/Course should note the subtree is untouched — the refusal is atomic, so nothing partial happened and no cleanup is owed. The Release/* records are also invisible to get @Edu/Course/* (which returns []), so the descendant enumeration and the child listing disagree about what is under that path; that discrepancy is worth its own look and may be the same underlying dangling-type condition as #2951.

Correction: the descendants are answerable, and the blocker is a permission gate — measured, not inferred

I set out to implement the fix proposed in the diagnosis above and stopped: its stated mechanism does not hold, and the proposed trigger condition would never fire for this subtree. Evidence below, all taken on memex.systemorph.com today (read-only apart from two sanctioned delete retries, both of which refused atomically — the subtree is intact, verified after).

1. The descendants' owning NodeTypes are framework-static, not `Edu/Course`

Edu/Course's 150 descendants are not 150 Release records. They are:

set	count	nodeType	owning NodeType
`Edu/Course/Release/*`	75	`Release`	`ReleaseNodeType` — static C#, `AddReleaseType()`
`Edu/Course/_Activity/*`	72	`Activity`	`ActivityNodeType` — static C#
(remainder)	~3	—	—

Neither type is compiled from Edu/Course's configuration lambda. Both are registered as static mesh nodes carrying an in-process HubConfiguration delegate, so NodeTypeEnrichmentHelpers.EnrichWithNodeType resolves them on its static-provider fast path and never touches Edu/Course. So "none of its 150 Release/* hubs can be configured" is not what is happening — and the proposed predicate ("the descendant's owning NodeType is not compilable") is false for every one of these 150 descendants, so the proposed branch would never be taken and the bug would survive the fix.

(The listing in the timeout message showed only Release/* paths because it prints pending.Take(10) ordered ordinal-ignore-case, and Release sorts before _Activity. That is why the composition of the set was not visible.)

2. The descendants DO answer — "150 of 150" is a cold-start cost, not a structural impossibility

Two delete Edu/Course runs, seconds apart:

Run 1 (cold): 150 of 150 descendant(s) of 'Edu/Course' did not answer ValidateDeleteRequest within 30s — reproducing the report exactly.
Run 2 (immediately after, hubs warm from run 1): Access denied: user 'rbuergi' lacks Delete permission on 'Edu/Course/_Activity/compile-202608061047144679f268cdd43da4e119c07e985d441ad9e'

Run 2 got an answer from every one of the 150 and reached a real verdict. So they are not unanswerable by construction; the count equalling the descendant count is not proof of a structural cause. Independently: get @Edu/Course/Release/20260720155947-mzH-99vj returns the node instantly (that read opens a SubscribeRequest to that node's own per-node hub via MeshNodeStreamCache) — and that exact path was in run 1's unanswered list.

What "150 of 150" actually measures: PreValidateDescendantsObs posts one ValidateDeleteRequest per descendant, each cold-activating that descendant's per-node hub, and waits for all of them under a single MeshOperationOptions.Timeout (30 s). The budget is shared N ways and the activation cost is O(N). Nothing in the stage bound scales with the fan-out width, so a wide cold subtree reports 100% silence and a warm one reports the truth. This is defect #1.

3. The real blocker is a delivery gate that ignores the node type's own access rule

Run 2's verdict is the actual reason Edu/Course cannot be deleted through the API.

ValidateDeleteRequest carries [RequiresPermission(Permission.Delete)]. AccessControlPipeline evaluates it through RequiresPermissionAttribute.GetPermissionChecks, whose default implementation is literally yield return (hubPath, Permission) — a raw Delete on the satellite's own path, folded by hub.CheckPermissionOutcome. It never consults INodeTypeAccessRule / NodeTypeAccessRuleSet; grep confirms those are consumed only by RlsNodeValidator and the delete pre-flight.

ActivityNodeType registers SatelliteAccessRule, which maps a satellite's Delete to Update on its MainNode — here Edu/Course. That is exactly the reasoning NodeDeletePreflight already documents for itself (#2913: "an Editor holds Update and not Delete — so an Editor could PUBLISH a satellite and then be refused when erasing it"). #2913 fixed the pre-flight; it did not fix the delivery gate in front of it. So the fan-out is refused on the 72 _Activity satellites by a check the node's own registered access rule says should pass. This is defect #2, and it is the one that actually blocks the delete.

It also explains the workaround. The Delete page's query-set mode (/{anchor}/Delete?q=…) deletes one path at a time via IMeshService.DeleteNode(path); each such delete is a leaf, so PreValidateDescendantsObs short-circuits at zero descendants and the satellite gate is never reached for the Release set. It removes the releases, not the activities.

What I did NOT find

I found no evidence for the catch-22 as stated (a NodeType whose compile fails making its own descendants unanswerable). It may well be real for a subtree whose descendants are instances of the broken type — Edu/Course has none (0 instances, per #2951) — but it is not what #3061 is.

Suggested shape, replacing the earlier one

Two independent changes, in this order:

Make the [RequiresPermission] gate resolve the effective (path, permission) pair through the registered INodeTypeAccessRule before folding, so a satellite's Delete is checked the way its type says it must be. Today the gate and NodeDeletePreflight answer the same question two different ways, and the gate wins because it runs first. Pin it with a test that an Editor (Update, no Delete) can delete a satellite through the fan-out.
Bound the fan-out per descendant, not per stage — the caller's single budget can never be the bound that names which descendant starved, which is precisely what A recursive delete's commit stage can stall for its whole budget — pre-flight fan-out and budget nesting are fixed and deployed; the commit stall is unmeasured on memex-cloud #1198 set out to fix; the nested rung exists (MeshOperationOptions.NestedTimeout) but only the descendant's storage read rides it. Note also that HandleValidateDeleteRequest bounds only that read: RunDeletionValidatorsObs runs unbounded, so a validator that never emits is silence with no rung at all. Separately, AccessControlPipeline's fold has no terminal — no timeout, no empty-completion branch — while RlsNodeValidator, riding the same fold, has three for exactly the shapes A starved security-policy read leaves CreateNodeRequest hanging for the full timeout instead of failing fast #1446 and A starved permission fold parks every [RequiresPermission] delivery until the caller's own timeout — AccessControlPipeline has no terminal, and the bounded twin attributes the same event correctly #2742 established; a stalled or empty fold there drops the delivery with no DeliveryFailure and no log line.

State of the subtree after my two retries: untouched — 75 Release + 72 Activity, all at their original versions. Both refusals took the atomic up-front branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPe8CH4mFU9E74HafWppqL

🚨 Retracting my earlier comment — the diagnosis above is WRONG

My comment #issuecomment-5507282012 claimed the cause was "the fan-out is addressed to each descendant's own hub, and those hubs are configured from Edu/Course's configuration lambda, which does not compile." That is false, and acting on it would have produced a fix that never fires. The measurement in the comment below it is correct; read that one.

Three things I got wrong, each checkable:

1. The 150 descendants are not 150 Release records. They are 75 Release + 72 _Activity + ~3. I took the timeout message's sample at face value — it prints pending.Take(10) ordered ordinal-ignore-case, and Release sorts before _Activity, so the visible ten were all releases. A truncated list is not a census, and I read it as one.

2. Their owning NodeTypes are NOT Edu/Course. ReleaseNodeType and ActivityNodeType are static C# in src/MeshWeaver.Graph/Configuration/, resolved by NodeTypeEnrichmentHelpers.EnrichWithNodeType on its static-provider fast path — Edu/Course is never consulted. So my proposed predicate ("the descendant's owning NodeType is not compilable") is false for all 150 and the branch would never be taken. I proposed a fix gated on a condition that never holds.

3. "150 of 150, never 149" is not structural. I argued a 100% non-answer had to be deterministic. It is a COLD-START cost: on a warm retry seconds later every one of the 150 answered — with a different and much more informative error (Access denied: user 'rbuergi' lacks Delete permission on 'Edu/Course/_Activity/compile-…'). My "deterministic by construction" reasoning was the strongest-sounding part of the comment and the most wrong.

The actual cause, which I have verified independently

ValidateDeleteRequest carries [RequiresPermission(Permission.Delete)], and the delivery gate evaluates it through AccessControlPipeline → attr.GetPermissionChecks(delivery, hubPath), which yields a raw Delete on the satellite's own path.

It never consults INodeTypeAccessRule. I confirmed the binding: NodeTypeAccessRuleSet is read in exactly two places — RlsNodeValidator and the delete pre-flight at MeshExtensions.cs:3348. AccessControlPipeline is not one of them.

ActivityNodeType registers SatelliteAccessRule, which maps a satellite's Delete to Update on its MainNode. #2913 fixed exactly that mismatch in NodeDeletePreflight — but not in the delivery gate that runs first. That is what refuses the 72 _Activity satellites.

It also explains the workaround, which my version did not: the Delete page deletes one path at a time via IMeshService.DeleteNode, each a leaf, so PreValidateDescendantsObs short-circuits at zero descendants — it removes the releases and never reaches the activities.

What is still true from the earlier comment

Only the band-aid list. Raising the 30s timeout, or treating a non-answer as "no objection", remain wrong answers — and the cold/warm measurement makes the first one actively misleading, since a longer bound would make cold runs appear to work while leaving the permission mismatch in place for anyone who hits it warm.

Prod is untouched: both refusals took the atomic up-front branch, verified afterwards at 75 Release + 72 Activity, all at original versions.

Merge pull request #3100 from Systemorph/fix/3061-delivery-gate-node-…

Fixed by #3100, merged at 11:13:29Z.

The cause

ValidateDeleteRequest carries [RequiresPermission(Permission.Delete)]. The delivery gate folds it through AccessControlPipeline → RequiresPermissionAttribute.GetPermissionChecks, whose default body is literally yield return (hubPath, Permission) — a raw Delete on the satellite's own path. It never consults INodeTypeAccessRule. NodeTypeAccessRuleSet was read in exactly two places, RlsNodeValidator and the delete pre-flight; the delivery gate was neither, and it runs first.

ActivityNodeType registers SatelliteAccessRule, which maps a satellite's Delete to Update on its MainNode. #2913 fixed that mismatch in NodeDeletePreflight and not in the gate ahead of it — which is why it survived.

AccessControl.md documented the gap as a deliberate non-decision: "Widening the message-level gate is a separate decision with a much larger blast radius; it has not been made." This issue falsifies that paragraph, and #3100 replaces it.

The fix

One NodeTypeAccessRuleGate seam owning both resolution and evaluation — the three terminals, the TakeDecisionOutsideGate hop, the "detail names the exception TYPE, never its message" rule. The pre-flight used to write all of that inline, which is precisely how the two seams diverged; both now call the one gate.

Only a denied check is re-decided. A grant is delivered untouched (a hot SubscribeRequest reads no node), an Undetermined fold stays Undetermined, Permission.Create maps to no operation, and a node read that faults answers Undetermined rather than the original denial — otherwise a storage blip would silently restore the old behaviour, which is the gate deciding its own input.

No timeout raised, no retry, no non-answer read as consent, no permission widened. Three verdicts pinned on one mesh, so a fix that merely widened Delete would fail two: Editor served, Viewer still refused Unauthorized, same Editor at a plain Markdown child still refused. 2479 tests across five full unfiltered projects.

🚩 Contradicting evidence, recorded rather than smoothed over

The production grant topology does not explain the reported denial, and this was not closed read-only. Edu/_Access holds only system-security, Public — Viewer, Anonymous — Viewer. rbuergi has no grant anywhere under Edu, no root _Access grant, zero GroupMembership nodes, and Edu/_Policy is { publicRead: true } — no cap, no breaksInheritance.

Yet the delete reached pre-validate-descendants, which only happens after the root Delete check on Edu/Course passed. And GetScopeHierarchy is a plain path walk, so any inheritable grant that let the root through would also have covered Edu/Course/_Activity/compile-….

So the grant that admitted the root is invisible to search — most plausibly a static assignment seeded into the deployment (CollectStaticAccessAssignments). One GetEffectivePermissions call on that portal would settle it. This changes nothing about the fix's correctness; the exact production trigger is not fully reconstructed, and saying so is better than implying it is.

Also corrected

An earlier comment of mine on this issue was wrong and is retracted above — I claimed the cause was per-node hubs unable to configure from a non-compiling NodeType. The measurement disproved it: the descendants are 75 Release + 72 _Activity, their owning types are static C#, and "150 of 150" is a cold-start cost, not structural (a warm retry answered all 150, with the permission denial named).

Secondary defects, reported not fixed: the gate's fold has no Timeout (a stall drops the delivery silently — the empty-completion half is covered, via CheckPermissionOutcome → Undetermined, #2742); PreValidateDescendantsObs bounds an O(N) cold fan-out with one stage budget; RunDeletionValidatorsObs is unbounded. Each needs its own MeshOperationOptions rung — the #1198 ladder — rather than being folded in here.

Closed by hand. Fifth time today the keyword registered the link and the issue stayed open: auto-merge lands a merge commit carrying only the PR title.

The pre-flight takes the node type's own second opinion, so waking it…

A session's permission check can actually fail, and a denial is an an…

Navigation Menu

Repository navigation

DeleteNode pre-validate-descendants: 150 of 150 Release records of an orphan NodeType never answer ValidateDeleteRequest — the orphan cannot be deleted #3061

What is failing

Why it is a defect, not a slow mesh

Where to look

Acceptance

The catch-22

Why the existing hardening cannot help

The fix shape (not implemented — deliberately)

Operationally, right now

Correction: the descendants are answerable, and the blocker is a permission gate — measured, not inferred

1. The descendants' owning NodeTypes are framework-static, not `Edu/Course`

2. The descendants DO answer — "150 of 150" is a cold-start cost, not a structural impossibility

3. The real blocker is a delivery gate that ignores the node type's own access rule

What I did NOT find

Suggested shape, replacing the earlier one

🚨 Retracting my earlier comment — the diagnosis above is WRONG

The actual cause, which I have verified independently

What is still true from the earlier comment

The cause

The fix

🚩 Contradicting evidence, recorded rather than smoothed over

Also corrected

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DeleteNode pre-validate-descendants: 150 of 150 Release records of an orphan NodeType never answer ValidateDeleteRequest — the orphan cannot be deleted #3061

Description

What is failing

Why it is a defect, not a slow mesh

Where to look

Acceptance

Activity

rbuergi commented on Sep 2, 2026

The catch-22

Why the existing hardening cannot help

The fix shape (not implemented — deliberately)

Operationally, right now

rbuergi commented on Sep 2, 2026

Correction: the descendants are answerable, and the blocker is a permission gate — measured, not inferred

1. The descendants' owning NodeTypes are framework-static, not Edu/Course

2. The descendants DO answer — "150 of 150" is a cold-start cost, not a structural impossibility

3. The real blocker is a delivery gate that ignores the node type's own access rule

What I did NOT find

Suggested shape, replacing the earlier one

rbuergi commented on Sep 2, 2026

🚨 Retracting my earlier comment — the diagnosis above is WRONG

The actual cause, which I have verified independently

What is still true from the earlier comment

rbuergi commented on Sep 2, 2026

The cause

The fix

🚩 Contradicting evidence, recorded rather than smoothed over

Also corrected

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. The descendants' owning NodeTypes are framework-static, not `Edu/Course`