Plan activation never settles after a paid checkout — buyer charged, plan not live, fallback reason hides the real cause

What is failing

A paid plan checkout is stamped Fulfilled, but the subscription activation the PaymentInboxWatcher filed never came back with a verdict: Subscriptions.ObserveActivationSettled resolved to an inactive outcome with no Error, so the watcher recorded its fallback ActivationDidNotSettle and the buyer is charged while their plan is not live. Money changed hands; the entitlement is missing until a human acts.

Probable cause

The log site is the non-active branch of PaymentInboxWatcher.ActivatePlan (Store Catalog source, Store/Catalog/Source/PaymentInboxWatcher). The activation request was filed via Subscriptions.RequestActivation on the record's own hub, but ObserveActivationSettled returned an outcome that was neither Active nor carried a control-plane error — i.e. the settle observation completed without a verdict (timeout / cold or disposed hub / sweep that never ran) and the code reports the fallback string instead of the underlying cause. Confidence: medium — the code path is confirmed, but the incident carries no exception or stack, so why the settle returned no verdict is not determinable from the log alone; it needs the subscription control plane's own logs for rbuergi at 2026-09-26T15:17:52Z.

Impact

One occurrence, one pod (memex-cloud portal), one buyer (rbuergi, personal plan, stripe subscription sub_1UJxLY1RJOWModnTvX8uv0YB, order rbuergi/_Orders/order-e2ab8e9eedca42c2a6765dce923e3ebb). The order is marked Fulfilled with the reason in Error, and the delivery is designed to be replayable from Stripe once the cause is fixed. One-off so far, but the class is "customer paid, nothing delivered" — worth a root-cause look at why ObserveActivationSettled can return a verdict-less outcome without surfacing why.

Where to look

PaymentInboxWatcher.ActivatePlan and the ActivationDidNotSettle fallback — Store/Catalog/Source/PaymentInboxWatcher
Subscriptions.RequestActivation / Subscriptions.ObserveActivationSettled — the settle observation that produced a verdict-less inactive outcome
The concrete order node rbuergi/_Orders/order-e2ab8e9eedca42c2a6765dce923e3ebb (stamped Fulfilled, Error carrying the fallback text)

No existing incident or issue covering this component and symptom was found in the mesh — I believe this is the first report of this class.

Evidence


Fingerprint	`52cff82c245e43d3`
Category	`PaymentInboxWatcher`
Severity	Error
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-6d84f5d754-4p9mg`
Occurrences	1
First seen	2026-09-26 15:17:52Z
Last seen	2026-09-26 15:17:52Z
Routing	not determined — no configured route matches the category `PaymentInboxWatcher`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-09-26 15:17:52Z memex-portal-deployment-6d84f5d754-4p9mg fail: PaymentInboxWatcher[0]
      [Payments] rbuergi PAID for the personal plan (order rbuergi/_Orders/order-e2ab8e9eedca42c2a6765dce923e3ebb, stripe subscription sub_1UJxLY1RJOWModnTvX8uv0YB) but the activation did not settle: Your payment went through, but activating the plan did not finish. Nothing more is owed — reload in a moment, and if it persists the Store team can activate it directly.. The order carries the reason; replay the delivery from Stripe once the cause is fixed.

_{Opened automatically from Admin/_LogIncident/52cff82c245e43d3. Recurrences are folded into this issue rather than opening new ones.}
_{It also stands for the whole log site c5b86fc8928155c0: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

Still occurring: 1 more since the last update (2 total).

Last seen: 2026-09-26 15:18:37Z
Pods: memex-portal-deployment-6d84f5d754-4p9mg

Recent log lines

2026-09-26 15:17:52Z memex-portal-deployment-6d84f5d754-4p9mg fail: PaymentInboxWatcher[0]
      [Payments] rbuergi PAID for the personal plan (order rbuergi/_Orders/order-e2ab8e9eedca42c2a6765dce923e3ebb, stripe subscription sub_1UJxLY1RJOWModnTvX8uv0YB) but the activation did not settle: Your payment went through, but activating the plan did not finish. Nothing more is owed — reload in a moment, and if it persists the Store team can activate it directly.. The order carries the reason; replay the delivery from Stripe once the cause is fixed.
2026-09-26 15:18:37Z memex-portal-deployment-6d84f5d754-4p9mg fail: PaymentInboxWatcher[0]
      [Payments] rbuergi PAID for the personal plan (order rbuergi/_Orders/order-e2ab8e9eedca42c2a6765dce923e3ebb, stripe subscription sub_1UJxLY1RJOWModnTvX8uv0YB) but the activation did not settle: Your payment went through, but activating the plan did not finish. Nothing more is owed — reload in a moment, and if it persists the Store team can activate it directly.. The order carries the reason; replay the delivery from Stripe once the cause is fixed.

This issue and Systemorph/MeshWeaver.Plugins#2405 describe the same incident (same order, same subscription). The root cause is established, and the fix is merged in Plugins, where this code lives: Systemorph/MeshWeaver.Plugins#2406 (merged 2026-09-26 16:10Z).

A correction to the report: the plan WAS live. Admin/Subscriptions/rbuergi went Done/Active at 15:17:09Z, three seconds after the delivery. What failed was the reading of that result, not the activation.

Root cause. Subscriptions.ReadActivation recognised state only as the string "Done". On a live mesh the record reaches the Store hub TYPED (SubscriptionContent). NodeElement.Of then round-tripped it with no string-enum policy, so state came out as the ordinal 2. The settle wait therefore never recognised a settled run. It lapsed after its 45 s budget and returned the verdict-less outcome, which the watcher logged as the fallback. The timings fit to the second: 15:17:06 + ~1 s hand-off + 45 s = 15:17:52, the log line above.

Second defect, worse than the log line. The "replay repairs a fulfilled order that carries an Error" branch then re-filed Activate for the same checkout. On a billed cadence that is a top-up, so one payment granted two months (the record logged two activations). The maintainer corrected the record by hand.

Fix (#2406):

ReadActivation reads the name or the ordinal, and NodeElement writes enum names.
The settle wait is correlated to its own request: a per-attempt activationRequestId, settled only when the action is cleared.
Active is read only from a successful Done.
Each paid activation carries its payment's grant key (checkout:{session}). The subscription control plane applies a key once and settles a duplicate without extending.

Renewals get the same exactly-once rule (invoice:{id}) in Systemorph/MeshWeaver.Plugins#2410.

Per policy severity-closes-on-verification I am not closing this. It closes once the Store package carrying #2406 is rolled onto memex-cloud and a test-mode plan checkout ends Fulfilled with no error and exactly one [active] line.

Production delivery check (2026-09-28 18:35Z, memex-cloud): Store/Catalog is compilationStatus: Ok, compiled at 14:33Z against the corrected PaymentInboxWatcher, Subscriptions, and NodeElement sources; the affected Admin/Subscriptions/rbuergi is Done/Active. The read-only payment-path audit ran on current-image pod memex-portal-deployment-59bb87c9d5-2qj79 (activity rbuergi/_Activity/643cf3cf6d2e4fa88563ec7caf0651de). Stripe is in TEST mode and the webhook endpoint is enabled, but the audit reports that endpoint lacks the required invoice.payment_failed event. This is a separate lifecycle gap; the checkout events needed for the happy-path witness are present. No fresh test-mode checkout/one-grant witness has been run, so this severity-H issue remains open. The next verification must use a test buyer and prove Fulfilled with no error plus one [active] line; do not replay the previously double-credited order as that witness.

Production adoption update (2026-09-30): the Plugins publication at 7b558192, which contains requested commit 90fbbcc5e, is sealed on the ci.9686 replicas. memex-cloud is 3/3 ready and Healthy on ci.9686 with that publication sealed. memex is still rolling with ci.9685/ci.9686 replicas and Degraded health, so adoption is not fleetwide there yet. Paid-plan activation remains unverified: the last documented audit found no fresh test checkout/one-grant witness and the Stripe test endpoint missing invoice.payment_failed. Keeping this open; an approved test buyer/flow is still needed, and no credentials should be posted in chat.

Confirmed the requested commit itself: Plugin Catalog CI run 36254530521 is for exact head 90fbbcc5e349e431893a0042f6c06ae92eecb216 and completed successfully. Both “Bake + publish NodeType assemblies to portal storage” and “Register the publication with memex” succeeded. Production adoption is visible on ci.9686 as noted above. This verifies publication/adoption only; it does not replace the missing paid-plan checkout witness.

Rollout adoption update, 2026-09-30 16:40Z: memex is now 3/3 ready on ci.9686 and all three replicas report Plugins 7b558192 sealed; memex-cloud is also 3/3 Healthy on ci.9686 with the publication sealed. memex health remains Degraded due bundle-adoption NoAssemblies misses and two previously broken NodeTypes; build is 1/1 ready but Degraded from bundle adoption as well. There is still no approved test-buyer checkout/one-grant witness, so paid-plan activation is unverified and this issue remains open.

Reopened by the fleet's bug-fix pool hand-over (maintainer request 2026-09-30: start a bug-fix thread for every open sev:H bug). This bug never got a bug-fix thread (it was recorded before the intake handed classified bugs to one, or never recorded); the reopened event is the intake's supported path to start it on the EU bug-fix pool. The close/reopen changes nothing else.

Publication/adoption recheck for commit 90fbbcc5e349e431893a0042f6c06ae92eecb216: Plugin Catalog CI run 36254530521 completed successfully, including the publish-bake job that registered the publication with memex and passed the inbox verification step. Current memex and memex-cloud ci9722 status reports show the MeshWeaver.Plugins publication at 5a1d850d sealed; GitHub compare confirms that commit descends from the target 90fbbcc5e.

Paid-plan activation is still not verified. I have not created or assumed a buyer identity or fixture. Please provide an approved production test-buyer/checkout fixture before that check; this issue stays open until a real successful paid checkout activates the plan.

Publication/adoption recheck, 2026-10-01 ~09:00Z: Plugin Catalog CI run 36254530521 remains successful for exact commit 90fbbcc5e349e431893a0042f6c06ae92eecb216. GitHub compare confirms published Plugins commit bf248649 is 925 commits ahead of that target with behind_by=0. Ops/Status/memex sampled 09:00:37Z is 3/3 ready on ci9727, and all three replicas report MeshWeaver.Plugins @bf248649 sealed. memex-cloud is in its automatic ci9724→ci9727 rollout: at the 09:00:41Z sample it was 3/3 ready but mixed (one ci9727, two ci9724), converged=false; the new ci9727 replica reports the publication sealed. The control-plane roll is still verifying, so I am not calling adoption fleetwide yet. No paid-plan checkout was run; an approved production test-buyer/checkout fixture is still needed from the owner, and this issue remains open until a real successful checkout proves activation.

Production recheck, 2026-10-05 11:57Z: run 36254530521 for target 90fbbcc5e349e431893a0042f6c06ae92eecb216 remains successful. The latest Plugins bundle 4c64b2c8 contains the target and is sealed for ci9983; some newly rolled ci9983 replicas report it sealed, while older 9965/9977 replicas report that same bundle unsealed. Both deployments are mixed-generation, not converged, and Degraded (memex 3/3 ready; memex-cloud 3/3 ready). This is partial adoption, not a successful paid-plan production check. No owner-approved checkout fixture was supplied or used. Leaving #5758 open.

Production recheck, 2026-10-05 12:56Z: the successful run for 90fbbcc5e349e431893a0042f6c06ae92eecb216 remains unchanged. The current Plugins bundle 29bfaefb includes that target (GitHub compare: ahead 1,925, behind 0) but is sealed only for platform ci9985, newer than the current ci9983 images, so latest-bundle adoption remains unverified. Memex-cloud is now Healthy 3/3 and converged; memex is converged 3/3 but still Degraded. No owner-approved checkout fixture has been supplied or used, so paid-plan activation is unverified and the issue stays open.

Production recheck, 2026-10-05 14:22Z: run 36254530521 for target 90fbbcc5e349e431893a0042f6c06ae92eecb216 remains successful, and latest bundle 29bfaefb contains it. Both memex and memex-cloud have converged 3/3 ready on ci9985 but report Degraded; their replicas say the bundle is sealed for ci9985 while the running framework is ci9984, so it is not adopted. Paid-plan activation remains unverified because no owner-approved checkout fixture has been supplied or used. Leaving #5758 open.

Production recheck, 2026-10-05 16:02Z: run 36254530521 for target 90fbbcc5e349e431893a0042f6c06ae92eecb216 remains successful. Both deployments still report the target-containing Plugins bundle @29bfaefb as NOT sealed for their running platform. Memex has regressed to 2/3 ready, Degraded and not converged; memex-cloud is 3/3 ready and converged but Degraded. Paid-plan activation remains unverified; no owner-approved checkout fixture has been supplied or used. Leaving #5758 open.

Residual root causes on the settle half are fixed in Systemorph/MeshWeaver.Plugins#3003. The incident itself was fixed in #2406/#2410.

Subscriptions.ObserveActivationSettled returned a fresh blank outcome on a lapse. It also returned a blank on a faulted read, and its own doc comment claimed otherwise. Every cause therefore logged the identical ActivationDidNotSettle line, which is the "fallback reason hides the real cause" of this issue. The lapse now carries the last reading: state/requestedAction/activationRequestId/error, or the read exception, plus the budget. The payment lane logs it after Cause:. The buyer-facing order text is unchanged.
SubscriptionControlPlane.Process skipped every Running emission. A record left Running by an activation that ended mid-run (a roll, a recycle, a crash) therefore never ran again, and every later paid activation on it lapsed. It is now resumed on the activation's first sight. This is exactly-once safe, because the grant key is written in the same terminal write.

This is not claimed as the 2026-09-26 cause; the ordinal read explains that one. Still open per severity-closes-on-verification: this closes on a post-roll test-mode checkout witness.

Production recheck on 2026-10-06: Plugin Catalog run 36254530521 succeeded for the exact requested commit 90fbbcc5e349e431893a0042f6c06ae92eecb216. The currently sealed Plugins publication @29bfaefb descends from that commit, and ci.10087 replicas report the bundle sealed/adopted. Rollout is still mixed and degraded: memex includes ci.10079 and ci.10087 pods; memex-cloud includes ci.10071 and ci.10087 pods. I found no owner-approved test-mode checkout fixture or paid-plan activation witness, so activation remains unverified and this issue stays open. User action needed: provide or approve the test-mode checkout fixture/identity for the post-roll production check; no live purchase or invented credentials were used.

Correction to my recheck comment: per-replica status confirms the Plugins publication @29bfaefb is sealed on ci.10087. The bundle_adoption field is aggregate and does not identify which publication supplied the adopted assembly, so I should not claim Plugins adoption was independently confirmed. Rollout remains mixed and degraded, and no paid-plan checkout witness exists. Please keep #5758 open pending an explicit adoption witness and the owner-approved test-mode checkout fixture.

Recheck: Plugin Catalog run 36254530521 completed for the exact requested head SHA 90fbbcc5e349e431893a0042f6c06ae92eecb216. The current published Plugins revision, 01f3db7c, is a descendant of that commit and is sealed on all current memex and memex-cloud replicas running ci.10114. This confirms the publication lineage, but not an artifact-specific adoption witness for the paid-plan path: bundle_adoption remains degraded and aggregate. No paid-plan activation check was run because no owner-approved test-mode checkout fixture is available. Leaving this open until both explicit adoption evidence and an approved checkout fixture support a successful production check.

Additional blocker from the automatic candidate × instance check, run 37586556277 (2026-10-07 07:19Z): the fleet roster derived successfully, but the required input assertion failed because COMBO_VERIFY_KEYS and COMBO_VERIFY_TOKENS were unset. The instance verification matrix was skipped; this is not a paid-plan activation result. I did not create identities or credentials. User direction and approved provisioning through the secret store are required before that check can run; leaving the issue open.

Read-only recheck at 10:32Z: Plugin Catalog run 36254530521 remains successful for requested SHA 90fbbcc5e349e431893a0042f6c06ae92eecb216. Current status samples show Plugins revision 01f3db7c sealed on the reported ci.10130 memex and memex-cloud replicas. This is publication-seal evidence, not a paid-plan activation result or artifact-specific adoption witness. The last candidate check (37586556277) failed its required-input assertion because COMBO_VERIFY_KEYS and COMBO_VERIFY_TOKENS were unset; no approved checkout fixture or later successful paid-plan check is evidenced. I created no identities or credentials; leaving the issue open pending user direction.

Production recheck (2026-10-07 14:25Z): Plugin Catalog run 36254530521 remains successful for exact commit 90fbbcc5e349e431893a0042f6c06ae92eecb216. Current per-replica status reports Plugins publication 01f3db7c sealed on the reported memex and memex-cloud rows; memex still has one unresolved provenance row and both deployments are Degraded. Paid activation is still unverified: latest candidate-by-instance run 37635868280 failed its required-input assertion and skipped the instance matrix. I did not invent a buyer or credentials. Owner action is still needed to provision an approved test buyer/checkout fixture and required secrets; keeping this open until a real paid checkout verifies activation.

Recheck on memex-cloud, 2026-10-07 ~16:05Z. Systemorph/MeshWeaver.Plugins#3003 (merged 02:09Z) is live: the sealed plugins publication is 01f3db7c on all 4 replicas on ci.10148, ahead of #3003's merge by 12 and behind by 0. The Store sources were written at 02:29Z, the Store NodeTypes were recompiled 13:00–13:14Z with 0 of 19 at compilationStatus:Error, and the replicas booted at 13:09Z or later.

With #3003, a lapsed settle names its cause (PaymentInboxWatcher.ActivationCause: refusal / settled-not-active / the record's last reading), and an orphaned Running record is resumed. #2406 fixed the original cause, the ordinal-state read. #2406 + #2410 fixed the double grant.

No plan checkout has run on memex-cloud since then. The only readable Store/Order across all declared partitions is still the 2026-09-26 one, so there is no live witness yet. This closes on a test-mode checkout ending Fulfilled with no error and one [active] line (procedure in Systemorph/MeshWeaver.Plugins#2405). Do NOT replay the old order: see the #2405 comment, where a replay would grant a third month because its record has no grant key.

Consolidated into https://github.com/Systemorph/MeshWeaver.Plugins/issues/2405, where the fix code lives. Evidence carried over in https://github.com/Systemorph/MeshWeaver.Plugins/issues/2405#issuecomment-6043113020.

Same root cause. Subscriptions.ReadActivation read the typed state as ordinal 2, so the settle wait lapsed into the ActivationDidNotSettle fallback, and the replay double-granted. This is the same order and subscription as fix(localmesh): install the AI engine so the shells' chat can start threads #2405.
Same fix. MeshWeaver.Plugins#2406 and Autocomplete.Test drops MeshWeaver.AI — keeping the import that only looks like one #2410. fix(ci): the arm lane merges with an App token, so main's push lanes run (#2916) #3003 makes a lapse name its cause.
Same closing condition. A fresh test-mode plan checkout on memex-cloud by a non-rbuergi buyer that ends Fulfilled with no error and exactly one [active] line. Do not replay the old order.

Folded in — a DIFFERENT fingerprint from the same log site.

This issue tracks the log site c5b86fc8928155c0 (PaymentInboxWatcher), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.


Folded fingerprint	`f5476dfb90c37fea`
Log site	`c5b86fc8928155c0`
Category	`PaymentInboxWatcher`
Severity	Error
Detail	`[{id}] signature verification FAILED for {path} — retaining at {path} A buyer may have PAID and be waiting: check Commerce:Stripe:WebhookSecret, then replay the delivery from Stripe.`
Namespace	`memex-cloud`
Pods	`memex-portal-deployment-547ffcbc64-mfw4g`
Occurrences	1
First seen	2026-10-12 11:08:01Z
Last seen	2026-10-12 11:08:01Z
Already folded here	1 fingerprint(s)
Incident	`Admin/_LogIncident/f5476dfb90c37fea`
Triage called it	PaymentInboxWatcher retained an order-naming Stripe webhook delivery whose signature did not verify (memex-cloud)

What triage wrote about this fingerprint

Observed

One delivery to the public hook POST /api/hooks/Store/Payments was stored in the inbox at Store/Payments/_Inbox/973c8980476a49cba8b8059e8e91bf4b, and when the Store hub's watcher (Store/Catalog/Source/PaymentInboxWatcher, ProcessEvent → Decide) asked the Stripe provider to verify it, the signature check did not pass: Decide returned Reject — a delivery with CanAuthenticate == true (a Commerce:Stripe:WebhookSecret IS configured), Authentic == false, that NamesWork (its body names an order, or a viewer and a plan — the shape of a real checkout.session.completed or a renewal, and exactly the shape retention exists to catch).
The watcher behaved as designed (Router-traffic detector reported every routing HOP as a violation #1069): the delivery was retained verbatim at Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b (with rejectedAt, rejectionReason, receivedSignature stamped) and removed from the inbox. Nothing in the watcher malfunctioned; the incident is the arrival of a per-instance, work-naming delivery that the configured secret could not authenticate.

Where and when

One pod of namespace memex-cloud, one delivery, processed as it arrived (not a backlog burst). It landed at 2026-10-12 11:08 UTC, minutes before this incident opened.
The delivery ARRIVED and was stored — which excludes the IRoutingGrain grain type not registered in Orleans cluster — all RouteMessage calls fail #1109 picture (no endpoint registered / target not allowlisted: nothing arrives and nothing logs). An endpoint for this instance's hook URL is delivering today.
It follows heavy payment-path reconfiguration in the preceding days: the Stripe package 1.2 landed 2026-10-11, the EnsurePaymentEndpoint maintenance task shipped with the ci: test runs are not dark — timestamped host start, live test output, long-running detection #2789 fix (PR fix(security): the API-token clamp now reaches message-routed checks (#2976) #3002, 2026-10-07), and Hosting/PaymentsApp was recycled repeatedly on 2026-10-06/07. Whether any of that touched this instance's endpoint or its vault secret was not established.

Impact

If the retained delivery is a genuine checkout or renewal, a buyer has paid and their order sits at PendingPayment with no entitlement and no install — and nothing else on the mesh would say why. The Store's own runbook treats a non-empty _Rejected as "someone may have paid and not been served": always a finding.
So far the scope is one event: no further rejections were seen after this one up to the incident's close.

Not established

The cause of the failed verification. The log site does not distinguish a wrong key from a stale timestamp from a forged body.
What the retained node contains. This identity could not read Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b: the container is a Store-admin-readable sensitive segment (CatalogGate.SensitiveSegments) and neither _Inbox nor _Rejected is visible through the search index — Store/Payments documents that both must be read directly. Its receivedSignature and body are the single most informative next read.
Whether this was a real purchase or a test-mode or junk POST with a well-formed body: the mode the portal's key charges in (TEST, per the IRoutingGrain grain type not registered in Orleans cluster — all RouteMessage calls fail #1109 record and the current Stripe/Guide) and the event named inside the body are both unread from here.
Whether rejections recur after 11:08.

Candidate causes and the reading that separates them

The configured Commerce:Stripe:WebhookSecret does not match the signing secret of the endpoint that sent this delivery (a rotation, a copy-paste slip, or a secret belonging to a different endpoint). Predicts: every subsequent delivery from that endpoint fails the same way, and the retained node's receivedSignature will not verify against the vault value but will against the endpoint's secret in the Stripe dashboard. Separating reading: compare the retained node's receivedSignature and body against the dashboard's delivery log for this event, and watch for further occurrences at this log site.
A second endpoint for the same URL, with a mismatched stored secret — the exact break-mode the pre-ci: test runs are not dark — timestamped host start, live test output, long-running detection #2789 audit remedy produced: a second endpoint for the same URL plus a swapped stored secret made every delivery from the existing endpoint fail verification and land in _Rejected. Predicts: the Stripe dashboard (in the mode this portal's key charges in) lists two endpoints for {Commerce:BaseUrl}/api/hooks/Store/Payments, and events arrive twice with at most one verifying. Separating reading: a PaymentPathAudit run (it hands a doubled listing to the provider and names it), or the dashboard listing; also the EnsurePaymentEndpoint / DeregisterPaymentEndpoint request nodes under Admin/Maintenance for this instance, and the vault value memexcloud-Stripe-WebhookSecret against the listed endpoints' secrets.
The delivery's Stripe-Signature timestamp fell outside the provider's five-minute replay window when it was processed. Predicts: the timestamp inside receivedSignature on the retained node is more than five minutes earlier than its rejectedAt. Separating reading: that comparison on the retained node.
A forged POST. The hook endpoint is public and unauthenticated, and Decide retains any well-formed body that names work. Predicts: no corresponding delivery in the Stripe dashboard's log for this instant. Separating reading: the dashboard's delivery log.

Where to look

Store/Catalog/Source/PaymentInboxWatcher — Decide, ProcessEvent (the Reject arm), RejectedNode.
The retained delivery at Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b — read directly by a Store admin, not via search.
The Stripe provider's verification (HMAC-SHA256 over {timestamp}.{body}, constant-time compare, five-minute window) in MeshWeaver.Payments.Stripe / StripePaymentProvider.
Related, not duplicates: Router-traffic detector reported every routing HOP as a violation #1069 (the retention this log shows working as designed), IRoutingGrain grain type not registered in Orleans cluster — all RouteMessage calls fail #1109 (endpoint absent — excluded here, the delivery arrived), Plugins#2789 (the audit remedy whose pre-fix consequence produces exactly this symptom; fixed by PR fix(security): the API-token clamp now reaches message-routed checks (#2976) #3002 on 2026-10-07).

Severity note: single delivery on one pod, recoverable by design (the delivery is retained verbatim and the fulfilment chain is idempotent under replay), cause not established and plausibly configuration rather than code — hence sev:M. A human should re-rule if the retained node shows a live-mode purchase, or if further rejections recur at this site.

Left closed — this ticket was closed as NOT PLANNED, which is a decision rather than a claim that the fault stopped. The occurrences below are recorded here; if the decision should change, that is a person's call and not a recurrence's.

Still occurring: 1 more since the last update (1 total).

Last seen: 2026-10-12 11:08:01Z
Pods: memex-portal-deployment-547ffcbc64-mfw4g

Recent log lines

2026-10-12 11:08:01Z memex-portal-deployment-547ffcbc64-mfw4g fail: PaymentInboxWatcher[0]
      [Payments] signature verification FAILED for Store/Payments/_Inbox/973c8980476a49cba8b8059e8e91bf4b — retaining at Store/Payments/_Rejected/973c8980476a49cba8b8059e8e91bf4b. A buyer may have PAID and be waiting: check Commerce:Stripe:WebhookSecret, then replay the delivery from Stripe.

Navigation Menu

Repository navigation

Plan activation never settles after a paid checkout — buyer charged, plan not live, fallback reason hides the real cause #5758

Evidence

32 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Plan activation never settles after a paid checkout — buyer charged, plan not live, fallback reason hides the real cause #5758

Description

Evidence

Activity

systemorph-com commented on Sep 26, 2026

rbuergi commented on Sep 26, 2026

rbuergi commented on Sep 28, 2026

rbuergi commented on Sep 30, 2026

rbuergi commented on Sep 30, 2026

rbuergi commented on Sep 30, 2026

rbuergi commented on Sep 30, 2026

rbuergi commented on Oct 1, 2026

rbuergi commented on Oct 1, 2026

32 remaining items

rbuergi commented on Oct 5, 2026

rbuergi commented on Oct 5, 2026

rbuergi commented on Oct 5, 2026

rbuergi commented on Oct 5, 2026

rbuergi commented on Oct 6, 2026

rbuergi commented on Oct 6, 2026

rbuergi commented on Oct 6, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

systemorph-com commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions