Skip to content

fix(layout): the Catalog guard reads the registry its render uses (#5065) - #5161

Merged
rbuergi merged 3 commits into
mainfrom
fix/5065-catalog-guard-reads-the-registry-the-render-uses
Sep 22, 2026
Merged

rbuergi merged 3 commits into
mainfrom
fix/5065-catalog-guard-reads-the-registry-the-render-uses

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Closes #5065.

The root: a guard that interrogated a different registry from the read

DomainLayoutAreas.Catalog resolved its ITypeDefinition from the hub's ITypeRegistry, then handed
it to DomainCatalogLayoutArea.RenderCatalog, which streams
Workspace.GetStream(new CollectionReference(collectionName)) — DataContext.DataSourcesByCollection.
Two different tables. The type registry is the hub's SERIALIZATION registry: DataContext.Initialize
registers every mapped type source into it, PolymorphicTypeInfoResolver adds any type the hub merely
serialises, and every content type a NodeType declares lands there. DataSourcesByCollection holds only
what a data source MAPS.

For the mapped types the two agree, which is why this passed review and every test. For Harness — a
NodeType content record with no data source — they do not, and the guard let it through to die three
frames deeper:

System.ArgumentException: Collections Harness are not mapped to any source.
   at MeshWeaver.Data.WorkspaceStreams.CreateWorkspaceStream[TReduced,TReference](…)  WorkspaceStreams.cs:38
   at MeshWeaver.Data.ReduceManager`1.ReduceStream[TReduced](…)
   at MeshWeaver.Layout.Domain.DomainCatalogLayoutArea.RenderCatalog(…)              DomainCatalogLayoutArea.cs:46

An unhandled throw out of a render reaches LayoutAreaHost's top-level Catch, which logs at Error
— the level that auto-files an incident — and renders the exception's own text. So an AUTHORING mistake
(@@Catalog/Harness naming a type that is not a workspace collection) was reported as an engineering
defect and the viewer was shown a sentence about the framework's collection map. AreaErrorClassifier
already demotes exactly this class (IsAccessDenied, IsNodeGoneNotFound → Warning + viewer copy); an
unmapped collection belongs in that company.

The fix asks the registry the read uses — area.Workspace.DataContext.GetTypeSource(collection) —
which is the same map keyed the same way (DataContext.Initialize builds TypeSources and
DataSourcesByCollection in one pass) and the set AddTypesCatalogs enumerates to OFFER catalog
links. The accept set and the offer set become one set, which is what stops the guard drifting from the
read a second time. Both of the area's declines are now localized; the missing-Id one was a hard-coded
English literal.

The control

TestCatalogOfATypeThatIsNotAWorkspaceCollection sits opposite the existing TestCatalog across
exactly one condition (DataRecord is mapped, UnmappedRecord is not; both are in the same
TypeRegistry, which is why the old guard could not tell them apart). Reverting the guard alone
reproduces the production reading in-process:

[Warning] [MeshWeaver.Data.Workspace] There were unmapped collections: UnmappedRecord
[Error]   [MeshWeaver.Layout.Composition.LayoutAreaHost] Rendering failed for area Catalog
Exception: System.ArgumentException: Collections UnmappedRecord are not mapped to any source.
   at MeshWeaver.Data.WorkspaceStreams.CreateWorkspaceStream … WorkspaceStreams.cs:line 38
viewer text: "⚠️ **This area failed to render.** ``` Collections UnmappedRecord are not mapped to any source. ```"
control (guard reverted) this branch
DomainLayoutServiceTest total 3, failed 1, succeeded 2 total 3, failed 0, succeeded 3

The assertion is on the CONTROL TYPE, deliberately: before the fix the area still produced a control
(the host's Catch renders an error panel), so "the area produced something" proves nothing.

Two diagnostics in the same family

Both named nothing a reader could follow up, which is the reason each issue is open rather than fixed.

#4597's remaining half. ObjectAsExtensions.LogRecoveryFailure dropped the what its callers
already pass and its two sibling report sites already print, so
As<MarkdownContent> could not recover value: JsonException had no referent — not the node, not the
partition, not which of the many writers of markdown content sent it. The incident's own review recorded
that as the blocker: "LogRecoveryFailure deliberately withholds content AND carries no node path, so
there is nothing to look up."
It now prints for {What}. Content and exception text stay withheld,
which is the load-bearing half of that contract; a node path is already in this file's other two log
lines and ContentAs<T> is documented as "this same conversion with the node's path in the diagnostics",
so no new class of information is exported.

🚨 And that is a finding about the test. ObjectAsPrivatePayloadTest passed the private text AS the
what argument and asserted the log did not contain it — which reads as "content is not published" and
actually measures "the caller-supplied subject is dropped". Those are different claims and only the
first is the accessor's contract. The two strings are now separate, and the test asserts both halves:
the content is absent AND the subject is present.

#5120's refusal. JsonSynchronizationStream.CreateSynchronizationStream printed
"No reducer defined for MeshNodeReference from MeshNodeReference" — the same type parameter twice
where the reduced type belonged, doubled space and all, and neither the hub that refused nor the
subscriber that asked. It now names all four. The condition is real and reachable, not impossible: a hub
configured with AddData() HAS a workspace, so Workspace.SubscribeToClient runs and reaches the
reducer lookup, but only AddMeshDataSource registers MeshNodeReference → MeshNode — so the mesh's
infrastructure hubs (portal/nodeops-{meshId}, portal/reads-{meshId}), which carry a workspace and
deliberately own no mesh node of their own, accept the subscribe and have nothing to reduce with.
Registering a reducer there would be the wrong fix (a CreateNodePermissionAttribute anchors at the
RECEIVING hub's path — that is why WithNodeOperationExecution must never land on a per-node hub), so
what the refusal has to make answerable is who asked an address with no node for its node. That is the
subscriber, and the subscriber is now in the message. #5120 stays open until the next occurrence
names its caller.

Verification

  • dotnet build -c Release -warnaserror — whole solution, 0 Warning(s) 0 Error(s), 66 assemblies,
    MeshWeaver.Hosting.dll rewritten inside the build window (so the dependents recompiled against the
    changed Data and Mesh.Contract).
  • DomainLayoutServiceTest 3/3 · ObjectAs* + LocalizationTest 80/80 ·
    NodeUpdateContentTypeChangeTest (NodeUpdatePipeline logs "As<MarkdownContent> could not recover value" at Error for stored content with no $type (EmailContent half fixed by #4679; remainder fixed in #5926) #4597's own control suite) 6/6 ·
    DocumentationLinkIntegrityTest 1/1 — which caught six bare sibling links in the new page first.
  • No public type or member removed, no interface member added, no behaviour change behind an unchanged
    signature reaching another repo. LogRecoveryFailure is private.
  • Nothing to recycle: no NodeType source, no node content shipped in an image or a bake, and no layout
    area compiled in the mesh is changed by this diff.

Mirror-sync: npm run sync:i18n -- --ref <merged core sha> in MeshWeaver.Plugins for the two added keys
catalog.needsType and catalog.notAWorkspaceCollection. Both are NEW keys with no value drift, so the
plugins drift guard is green either way — this is the debt-declaration, not a red.

Doc: Doc/Architecture/ThreeRegistriesAReadCanMiss, linked from the Architecture topic map.

🤖 Generated with Claude Code

)

`DomainLayoutAreas.Catalog` resolved its `ITypeDefinition` from the hub's
`ITypeRegistry` and then handed it to a render that streams
`Workspace.GetStream(new CollectionReference(...))` — it checked one registry and
read another. The two are not the same set and are not meant to be: the type
registry is the hub's SERIALIZATION registry, which every content type a NodeType
declares lands in, while `DataContext.DataSourcesByCollection` holds only the
collections a data source MAPS.

A type in the first and not the second walked past the guard and died three frames
deeper as a bare `ArgumentException: Collections Harness are not mapped to any
source` — an unhandled throw out of a render, so `LayoutAreaHost` logged it at
Error (the level that auto-files an incident) and put that framework sentence in
the viewer's face for what is an authoring mistake.

The guard now asks `DataContext.GetTypeSource(collection)`, which is the same map
keyed the same way AND the set `AddTypesCatalogs` enumerates to OFFER catalog
links — so the accept set and the offer set are one set. Both of the area's
declines are now localized (`catalog.needsType`, `catalog.notAWorkspaceCollection`);
the first was a hard-coded English literal.

Control: `TestCatalogOfATypeThatIsNotAWorkspaceCollection` sits opposite the
existing `TestCatalog` across exactly one condition. Reverting the guard alone
reproduces the production reading in-process — `There were unmapped collections:
UnmappedRecord`, `Rendering failed for area Catalog` at Error,
`WorkspaceStreams.cs:38` — and fails the new test; 1 failed / 2 succeeded against
3 succeeded.

Two diagnostics in the same family, both of which named nothing a reader could
follow up:

* `ObjectAsExtensions.LogRecoveryFailure` dropped the `what` its callers already
  pass and its two sibling report sites already print, so `As<MarkdownContent>
  could not recover value: JsonException` had no referent — not the node, not the
  partition, not which writer sent it. That is #4597's remaining half, and its
  review recorded the missing subject as the blocker. Content and exception text
  stay withheld, which is the load-bearing part of that contract.
  `ObjectAsPrivatePayloadTest` passed the private text AS the subject and asserted
  its absence, which reads as "content is not published" but measures "the
  caller's subject is dropped" — the two strings are now separate and the test
  asserts both halves.
* `JsonSynchronizationStream`'s reducer refusal printed the same type parameter
  twice where the reduced type belonged, and named neither the hub that refused
  nor the subscriber that asked (#5120). It now names all four. The condition is
  real and reachable: a hub with `AddData()` has a workspace and reaches the
  lookup, but only `AddMeshDataSource` registers `MeshNodeReference -> MeshNode`,
  so the mesh's infrastructure hubs — which own no node on purpose — accept the
  subscribe and have nothing to reduce with.

Doc: `Doc/Architecture/ThreeRegistriesAReadCanMiss`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes consistently align guard behavior with the actual workspace registry used by the render path, add localized viewer-facing copy, and include targeted regression coverage and clearer diagnostics without introducing async/task patterns in src/.

Review effort: Lite
Findings: None

What changed in this PR

This PR fixes a production-facing failure mode in the Domain @@Catalog layout area by ensuring its precondition guard interrogates the same registry that the render path actually reads (workspace-mapped collections), preventing unmapped-but-serializable types (e.g., NodeType content records) from reaching Workspace.GetStream(CollectionReference) and throwing during render.

Changes:

  • Align DomainLayoutAreas.Catalog guard with the workspace DataContext collection/type-source map and return localized, actionable cautions instead of throwing for unmapped collections.
  • Strengthen diagnostics: include the caller-supplied subject in ObjectAsExtensions recovery-failure logs, and expand reducer-missing errors in JsonSynchronizationStream to name hub/subscriber/types/stream.
  • Add regression tests and an architecture doc page explaining the three distinct “not registered” registries involved in these failures.
File Description
test/​MeshWeaver.Messaging.Hub.Test/​ObjectAsPrivatePayloadTest.cs Updates test to distinguish “private content” from diagnostic subject and assert both privacy + diagnosability.
test/​MeshWeaver.Layout.Test/​DomainLayoutServiceTest.cs Adds a negative control covering “type is in TypeRegistry but not a workspace collection” behavior.
src/​MeshWeaver.Messaging.Hub/​Localization/​strings.en.json Adds localized catalog error/help strings (English).
src/​MeshWeaver.Messaging.Hub/​Localization/​strings.de.json Adds localized catalog error/help strings (German).
src/​MeshWeaver.Mesh.Contract/​ObjectAsExtensions.cs Ensures recovery-failure logs include the subject (what) while still withholding content/exception messages.
src/​MeshWeaver.Layout/​Domain/​DomainLayoutAreas.cs Fixes Catalog guard to use workspace DataContext.GetTypeSource and return localized cautions instead of throwing.
src/​MeshWeaver.Documentation/​Data/​Architecture/​ThreeRegistriesAReadCanMiss.md Adds a new architecture page documenting the three registries and the failure modes.
src/​MeshWeaver.Documentation/​Data/​Architecture.md Links the new architecture page from the Architecture topic map.
src/​MeshWeaver.Data/​Serialization/​JsonSynchronizationStream.cs Improves “no reducer defined” refusal message to include hub/subscriber/types/stream context.
src/​MeshWeaver.Compiler.Pipeline/​NodeTypeCompilationHelpers.cs Updates commentary to reflect the improved reducer-missing diagnostic wording.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

429 tests   - 14   429 ✅  - 14   1m 2s ⏱️ +14s
  2 suites  -  1     0 💤 ± 0 
  2 files    -  1     0 ❌ ± 0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

  1 files  ±0    1 suites  ±0   2m 45s ⏱️ +4s
306 tests ±0  306 ✅ ±0  0 💤 ±0  0 ❌ ±0 
308 runs  ±0  308 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

533 tests   - 1 057   533 ✅  - 1 057   1m 3s ⏱️ - 2m 2s
  1 suites  -     1     0 💤 ±    0 
  1 files    -     1     0 ❌ ±    0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

    2 files   -   1      2 suites   - 1   5m 43s ⏱️ -13s
2 526 tests +591  2 526 ✅ +591  0 💤 ±0  0 ❌ ±0 
2 527 runs  +591  2 527 ✅ +591  0 💤 ±0  0 ❌ ±0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 550 tests  +830   1 550 ✅ +1 022   7m 38s ⏱️ + 2m 5s
    2 suites  -   1       0 💤  -   192 
    2 files    -   1       0 ❌ ±    0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    5 files  ±  0      5 suites  ±0   14m 2s ⏱️ -47s
3 230 tests  - 632  3 230 ✅  - 630  0 💤  - 2  0 ❌ ±0 
3 234 runs   - 632  3 234 ✅  - 630  0 💤  - 2  0 ❌ ±0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

   13 files   -   4     13 suites   - 4   32m 17s ⏱️ -37s
8 574 tests  - 282  8 574 ✅  - 88  0 💤  - 194  0 ❌ ±0 
8 581 runs   - 282  8 581 ✅  - 88  0 💤  - 194  0 ❌ ±0 

Results for commit baf2e26. ± Comparison against base commit b9d7fa7.

♻️ This comment has been updated with latest results.

@rbuergi

rbuergi commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

CI is red on one shard, and the failure is UNATTRIBUTED — this diff cannot reach it. Not merging, and not re-running.

Run tests (shard 5) → ANodeTypesSourcesWaitForItsBundleTest.AnAdoptedTypesSources_WaitForABundleThatCarriesThem
(run 35664055619, job 106546756733, 22:53:45.790Z). 3 854 tests, 3 851 passed, 1 failed; the other
five shards are green, and so is every other check including Automatic review answered.

MeshWeaver.Reactive.Assertions.AssertionException :
  Expected value to be AdoptedVerified because the type goes AdoptedVerified → AdoptedVerified,
  never through StaleAdopted, but found Compiled.

Everything the test is ABOUT passed; only BuildProvenance is wrong

22:53:45.488  live fingerprint at aa11aa22: 8ac5e69cec9266f9
22:53:45.668  seed outcome: Adopted
22:53:45.690  [BundleHold] Heldbc5b0306: 1 NodeType(s) HELD — … no bundle for framework identity
              gccd825b5362eecaeafac5fac8f8b8dbf4d5b5276 carries …
22:53:45.775  after the held import: commit=aa11aa22… outcome=Imported held=1
22:53:45.775  held: Heldbc5b0306/Widget has 8ac5e69cec9266f9, waits for 2dc42f3433a286fb
22:53:45.790  === TEST FAILED: … but found Compiled.

The seed adopted, the per-type hold fired, the held type kept its fingerprint, the import reported
held=1. The one assertion that failed is the provenance field, which read Compiled where the adoption
had just written AdoptedVerified — a local compile overwriting the adoption record after the seed. That
is a race in the compile-watcher / PrebuiltAssemblySeeder ordering, not in the hold this test measures.

Why this diff cannot reach it

Every production-code change in this PR, exhaustively:

Change Reachability
DomainLayoutAreas.Catalog guard only when a Catalog layout area renders; this test renders none
ObjectAsExtensions.LogRecoveryFailure one log message gains a format argument. Same level, same call count, identical return value and control flow
JsonSynchronizationStream refusal message text. Same condition, same throw
NodeTypeCompilationHelpers.cs a comment — the only file touched in the compile area, verified with git diff; no code changed
two localization keys, two test files additive

Nothing changes control flow anywhere near adoption, compilation, bundles or provenance.

Controls

  • The same test, locally, Release, same filter: total: 1, succeeded: 1. Passes on this branch.
  • main is green — the last eight MeshWeaver Build and Test runs are completed/success,
    including e3509fcf at 22:00Z, which is this branch's merge base.
  • Not in .github/known-flakes.json, so the merge-queue steward will not re-queue it — it would leave
    the PR out with queue-rejected. That is the right behaviour and I am not going to route around it.
  • No open issue mentions both AdoptedVerified and Compiled.

What I am NOT doing, and why

  • Not re-running the shard. Re-running to see whether it passes the second time is exactly what hides
    a race, and this repo's rule is explicit about it.
  • Not adding a flake-catalogue entry. That file is for "a TEMPORARY, EVIDENCE-BEARING allowance for
    one known race while its root cause is fixed — never a permanent tolerance"
    . I have not root-caused
    this, so an entry would be the tolerance the file forbids.
  • Not merging. CI is red.

Filed, with the evidence

The race is filed into bug triage on the control instance (a Feedback/Feedback submission, handed over
to the triage agent). What I did not establish is in the filing: which writer set Compiled, how
often it races, and whether it is CI-specific — one local pass is not a negative result.

What this PR needs: either that race fixed, or a decision by someone who owns the adoption path that
it is a known flake worth cataloguing. The change set itself is complete and verified.

@rbuergi

rbuergi commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

The handle for whoever picks this up: the assertion is a POINT READ, and it is stricter than its own stated reason

Two details from ANodeTypesSourcesWaitForItsBundleTest that decide whether this is a test race or a real
adoption defect — and I am deliberately not deciding it, because the answer changes what #3845's
acceptance criterion means:

// line 146 — a WAIT. At this point the provenance genuinely IS AdoptedVerified.
var adopted = await DefinitionWhen(d => d.BuildProvenance is BuildProvenance.AdoptedVerified, ct);

// … the held reimport at commit B …

// line 182 — a POINT READ of whatever the record says right now, with nothing to filter on.
var stillAdopted = await Definition(ct);
// line 184
stillAdopted.BuildProvenance.Should().Be(BuildProvenance.AdoptedVerified,
    "the type goes AdoptedVerified → AdoptedVerified, never through StaleAdopted");
  1. It is an un-waited snapshot asserting a negative. Line 146 waits for the state; line 182 takes
    whatever is there. Any intermediate write between them is observable, so the assertion can fail on a
    state the system passes through rather than settles in.
  2. It failed on Compiled, which its own because does not mention. The justification is "never
    through StaleAdopted" — and Compiled is neither AdoptedVerified nor StaleAdopted. So either
    Compiled is also a violation (a held, adopted type was recompiled locally — a real defect, and the
    assertion is right but under-explained), or it is an acceptable transient the assertion should admit
    (a test race, and the fix is the assertion).

Both candidate fixes are wrong if the other reading is true: widening the assertion to admit Compiled
hides a real recompile of a held type, and suppressing the compile changes the adoption path. That is a
call for whoever owns #3845 hole 4, not for a PR that touches none of it.

Everything else on this PR is green, including Automatic review answered. Auto-merge is armed and will
fire the moment the build is green.

@rbuergi

rbuergi commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

🚨 Correction: the adoption race did NOT reproduce on the new base. The second red was an artifact 403, and no test ran at all.

I merged main (base was fd71de62 → 35a4dfaa) and pushed db6495e7. That run also went red, but on a
different shard and for a different reason, and the distinction matters both ways.

ANodeTypesSourcesWaitForItsBundleTest passed on the new tree

Shard 5 was green on db6495e7. The only failing check was Run tests (shard 3). So the
AdoptedVerified → Compiled provenance failure did not recur on a second, independent tree — which
is the opposite of the "much stronger evidence" case, and I am saying so as plainly as I would have said
the reverse. One occurrence remains one occurrence.

Shard 3 did not fail a test — it never ran one

Test Results (shard 3):  0 tests  − 443     0 ✅ − 443     0s ⏱️ −41s
                         0 suites −   3     0 ❌ ±   0

0 tests, down 443 from the previous commit, and 0 ❌. The job lived 34 seconds (23:15:19Z →
23:15:53Z) and ended with:

##[error]Unable to download artifact(s): Failed to ListArtifacts: Received non-retryable error:
         Failed request: (403) Forbidden: Error from intermediary with HTTP status code 403 "Forbidden"
…
grep: test/test-results.log: No such file or directory

It could not download build-output-3 (ID 10668789114, 33.8 MB, digest recorded), so it had nothing to
test, wrote no test-results.log, and the shard's own collected-nothing gate reddened it correctly —
that gate exists precisely so "the shard ran no tests" cannot read as green. The gate did its job; the
input was missing.

A bare 403 from "an intermediary" names no cause, and this is a known trap here: an artifact-storage
refusal arrives wearing a workflow-defect costume and gets attributed to whatever diff happens to be in
flight. Two things say it is not this diff:

  • the failure is in actions/download-artifact, before any test binary is loaded;
  • it landed at 23:15:42Z, inside the same minute a shared account-level limit killed every agent in
    this session. An account/infrastructure refusal on artifact listing and a session-wide limit in the same
    minute is one event, not two.

What I did about it

Merged main again — it had advanced to b9d7fa70 (five PRs: #5157, #5158, #5159, #5160, #5162) — and
pushed baf2e26d. Clean merge, no conflicts, no lock or manifest file in the diff so nothing to
regenerate and nothing to revert (no moduleVersion restamped over an unchanged tree). Solution builds
0 Warning(s) 0 Error(s) with -warnaserror, 46 s of real work.

This is again a new tree, not a re-run: different base, different commit, never measured. I have not
re-run any shard, added nothing to known-flakes.json, and touched no assertion. Auto-merge stays armed.

@rbuergi

rbuergi commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

The third red is the SAME artifact 403, now on UPLOAD — and re-running it is the named exception, not a flake-hunt

baf2e26d (base b9d7fa70) went red on Build solution (once), and the compile was not the problem.
Artifacts build-output-0 … build-output-4 uploaded and finalized fine; the sixth did not:

02:40:16.5  Uploading artifact: build-output-5.zip
02:40:19.3  Finished uploading artifact content to blob storage!
02:40:19.3  SHA256 digest of uploaded artifact is 55101ccd4bfd5cde…
02:40:19.3  Finalizing artifact upload
02:40:19.8  ##[error]Failed to FinalizeArtifact: Received non-retryable error:
            Failed request: (403) Forbidden: Error from intermediary with HTTP status code 403 "Forbidden"

Content uploaded, digest computed, finalize refused. Paired with the earlier red on db6495e7
(ListArtifacts → the identical 403, 23:15:42Z), that is the same refusal on both ends of the artifact
plumbing, three hours apart. A 403 from "an intermediary" names no cause, which is exactly how an
artifact-storage refusal gets attributed to whichever diff is in flight.

Why re-running is allowed here and re-running the shard was not. The rule bans re-running a test to
see whether a flake passes, because that hides a race — and it names one exception: the harness itself
crashed.
A build job whose artifact finalize is refused by an external service never produced a test
result to paper over; there is no assertion whose outcome is being resampled. The repo's own merge-queue
steward uses the same taxonomy, re-queueing on evidence that includes an infrastructure death. So I
re-ran the failed jobs of run 35680084662 (now attempt 2). I have still not re-run
Run tests (shard 5), added nothing to known-flakes.json, and touched no assertion.

What I could not establish: whether this is fleet-wide. My run is the only one in this repository
since 23:49Z
, so there is no concurrent control. It is intermittent rather than a hard wall — the queue
and main runs between 23:24Z and 23:49Z all succeeded, after the 23:08–23:15Z failures. If attempt 2 fails
the same way, that is an account-level artifact-storage problem that will block every PR here, and it
needs a person rather than another run.

Standing tally on this PR, for whoever reads it next:

Head Red on Cause Reached by this diff?
4d996bdb Run tests (shard 5) AdoptedVerified → Compiled provenance race no — filed twice
db6495e7 Run tests (shard 3) artifact download 403; 0 tests collected no
baf2e26d Build solution (once) artifact upload finalize 403 no

Shard 5 passed on db6495e7, so the provenance race did not reproduce on a second tree.

@rbuergi
rbuergi added this pull request to the merge queue Sep 22, 2026
Merged via the queue into main with commit 744f0fe Sep 22, 2026
51 of 53 checks passed
@rbuergi
rbuergi deleted the fix/5065-catalog-guard-reads-the-registry-the-render-uses branch October 10, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Domain Catalog area render aborts: Harness collection not mapped to any workspace source

2 participants