Repository navigation
PII redaction filter on the model gateway blocks agent chat requests outright when its redaction context is lost, aborting agent rounds mid-flight #5932
Description
What is failing
The PII redaction layer in front of the model gateway rejects an outbound chat-completion request from MeshWeaver.AI.AgentChatClient outright with HTTP 403 (Request blocked: PII detected (redaction_context_lost)) instead of redacting and forwarding it. The ClientResultException surfaces inside the agent's streaming chat call and aborts the entire agent round — in the observed case a pr-reviewer round that had already completed its analysis (~29k tokens of output) and never got to post its review for Systemorph/MeshWeaver.Plugins#2619.
Probable cause
The gateway-side redaction filter could not find or restore its per-request redaction context — the reason code redaction_context_lost says the filter lost the state it needs to mask PII, and its fail-closed policy is to block rather than send unredacted or redact statelessly. Moderate confidence on the mechanism (the 403 body names it directly); low confidence on why the context was lost. One circumstantial hint: the blocked call came late in a long-lived round (after a ~9-minute model call and a failed patch confirmation), so expiry/eviction of redaction context on long rounds is a plausible trigger — but the evidence does not establish this.
Impact
A single occurrence on one pod so far, but the failure mode is expensive when it fires: the whole agent round is lost, including tool work already done, and the round ends in Error with nothing written back. For the PR-review pipeline that means a review silently not posted. Workaround: re-run the round. Severity left at M given one occurrence; if this recurs, it degrades every long agent round and should be re-ruled.
Where to look
The content-filter/redaction middleware on the model gateway serving z-ai/glm-5.3, on the request path behind MeshWeaver.AI.AgentChatClient → OpenAI ChatClient.CompleteChatAsync (top frame OpenAI.ClientPipelineExtensions.ProcessMessageAsync). Start from the code that emits the redaction_context_lost reason code: where redaction context is keyed, stored, and looked up per request, and what makes the lookup miss. Related known issue, same layer but a different failure mode (masking/corruption instead of blocking): Systemorph/MeshWeaver.Plugins#2567 — not a duplicate, but the same component and worth fixing together.
Evidence
| Fingerprint | 367de820103e2273 |
| Category | MeshWeaver.AI.AgentChatClient |
| Severity | Error |
| Exception | System.ClientModel.ClientResultException |
| Top frame | OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options) |
| Namespace | memex |
| Pods | memex-portal-deployment-747567ff6d-zsvvw |
| Occurrences | 1 |
| First seen | 2026-09-30 18:34:21Z |
| Last seen | 2026-09-30 18:34:21Z |
| Routing | not determined — no configured route matches the category MeshWeaver.AI.AgentChatClient. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-30 18:34:21Z memex-portal-deployment-747567ff6d-zsvvw fail: MeshWeaver.AI.AgentChatClient[0]
[ThreadExec] ERROR: 18:34:21.177 threadPath=Hosting/Triage/_Thread/review-pull-request-systemorph-meshweave-fa96
System.ClientModel.ClientResultException: HTTP 403 (: )
Request blocked: PII detected (redaction_context_lost)
at OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
at OpenAI.Chat.ChatClient.CompleteChatAsync(BinaryContent content, RequestOptions options)
at OpenAI.Chat.ChatClient.<>c__DisplayClass20_0.<<CompleteChatStreamingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at OpenAI.AsyncSseUpdateCollection`1.GetRawPagesAsync()+MoveNext()
at OpenAI.AsyncSseUpdateCollection`1.GetRawPagesAsync()+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at System.ClientModel.AsyncCollectionResult`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
at System.ClientModel.AsyncCollectionResult`1.GetAsyncEnumerator(CancellationToken cancellationToken)+MoveNext()
at System.ClientModel.AsyncCollectionResult`1.GetAsyncEnumerator(CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Microsoft.Extensions.AI.OpenAIChatClient.FromOpenAIStreamingChatCompletionAsync(IAsyncEnumerable`1 updates, ChatCompletionOptions options, CancellationToken cancellationToken)+MoveNext()
at Microsoft.Extensions.AI.OpenAIChatClient.FromOpenAIStreamingChatCompletionAsync(IAsyncEnumerable`1 updates, ChatCompletionOptions options, CancellationToken cancellationToken)+MoveNext()
at Microsoft.Extensions.AI.OpenAIChatClient.FromOpenAIStreamingChatCompletionAsync(IAsyncEnumerable`1 updates, ChatCompletionOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
…[truncated]
Opened automatically from Admin/_LogIncident/367de820103e2273. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 9f4ad2732d4debf3: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
systemorph-com commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
9f4ad2732d4debf3(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.828094ade6813b769f4ad2732d4debf3MeshWeaver.AI.AgentChatClientSystem.ClientModel.ClientResultExceptionOpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)[ThreadExec] ERROR: {n}:{n}:{n} threadPath={path}memexmemex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-85469d6f4b-lrxh7,memex-portal-deployment-85469d6f4b-2gwtz,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-65b5c866cd-mwb67,memex-portal-deployment-65b5c866cd-szgkhAdmin/_LogIncident/828094ade6813b76What triage wrote about this fingerprint
What is failing
MeshWeaver.AI.AgentChatClientfails agent-thread turns with OpenRouterHTTP 402 (ClientResultException). Every failing request sendsmax_tokensequal to the model's full output ceiling (131072 or 65536 tokens, per the model in use), and OpenRouter rejects it with "This request requires more credits, or fewer max_tokens" once the workspace key's remaining daily credit budget is lower than that pre-booked reservation. The failure is thrown mid-stream insideOpenAIChatClient.FromOpenAIStreamingChatCompletionAsyncand surfaces as an ERROR on the[ThreadExec]loop, killing the turn.Probable cause
High confidence. The exception message is self-describing and consistent across all samples: the request's
max_tokensexceeds the key's remaining daily credit budget, so this is not a transient outage. The defect is that AgentChatClient appears to setMaxOutputTokensto the model's maximum (or leaves it unset so the SDK/OpenRouter defaults to the full ceiling) instead of a bounded, configured value — so a conversation whose actual completion needs a few thousand tokens still tries to reserve 65k–131k and is refused whenever the daily budget has fallen below that number. This turns "budget is running low" into "all AI execution is dead until the daily limit resets". A secondary aspect: there is no fallback or retry with a reducedmax_tokenson 402, and no budget-aware cap.Impact
34 occurrences over ~9.5 days (2026-09-21 → 2026-10-01) across all 7 portal pods, still occurring as of the last sample. When it hits, every AI-thread execution on the shared portal fails — triage threads, Hosting bug-fix threads, and user chat alike — until the OpenRouter daily budget resets or is raised. This is a hard outage of the agent subsystem during those windows, though self-healing daily.
Where to look
The chat-client configuration where
MaxOutputTokens/max_tokensis set for the OpenRouter-backed model tier (log categoryMeshWeaver.AI.AgentChatClient, reached viaOpenAIChatClient.FromOpenAIStreamingChatCompletionAsync). Fixes to consider: capMaxOutputTokensto a sane configured default instead of the model ceiling, retry once with a reducedmax_tokenson HTTP 402 (the message states the affordable amount), and surface a budget-exhausted error to the thread instead of a generic 402.Workaround
Raise the OpenRouter key's daily credit limit (the error carries the workspace/keys URL), or configure a lower
max_tokensfor the affected model tiers. Note the API key id is partially exposed in the error body and may warrant rotation.Still occurring: 34 more since the last update (34 total).
memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-85469d6f4b-lrxh7,memex-portal-deployment-85469d6f4b-2gwtz,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-65b5c866cd-mwb67,memex-portal-deployment-65b5c866cd-szgkhRecent log lines
Re-addressed by the current identity function: this incident inherited
11973e8dfa3d0711(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
9f4ad2732d4debf3(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.0b9d6eb8fda3298a9f4ad2732d4debf3MeshWeaver.AI.AgentChatClientSystem.ClientModel.ClientResultExceptionOpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)HTTP {n} (: ) Provider returned errormemexmemex-portal-deployment-65b5c866cd-6bkwhAdmin/_LogIncident/0b9d6eb8fda3298aWhat triage wrote about this fingerprint
What is failing
During agent thread execution, the model provider (OpenRouter) rejected a streaming chat-completion request with HTTP 429 ("Provider returned error", no
Retry-Afterheader).MeshWeaver.AI.AgentChatClienthas no handling forSystem.ClientModel.ClientResultExceptionaround the streaming enumeration, so a transient rate-limit response kills the whole agent round with a raw stack trace instead of being retried or surfaced as a domain result.Probable cause
The top application frame (
OpenAI.ClientPipelineExtensions.ProcessMessageAsync) shows the failure on the raw HTTP transport boundary of the OpenAI SDK; the exception then escapesChatClient.CompleteChatStreamingAsync→OpenAIChatClient.FromOpenAIStreamingChatCompletionAsyncup to the generic[ThreadExec] ERRORsink. The defect is the missing catch-and-map inAgentChatClient: a 429 should trigger bounded exponential backoff and retry, not round failure. Confidence: high on what escapes (the stack is unambiguous and identical to the already-filed cases); medium on the retryability of this specific response — the provider body is generic and carries noRetry-After, so it may be a hard upstream capacity limit rather than a classic rate limit.Impact
Single occurrence on one pod — noise-level volume, not an outage. But each occurrence is a completely failed agent round with no graceful recovery or user-visible message, and the affected threads are the automated triage pipeline's own review threads. Context worth noting: the same pod was grinding through HTTP 402 credit-exhaustion failures from the same OpenRouter workspace within the preceding hour (incident
Admin/_LogIncident/828094ade6813b76, 45 occurrences) — this 429 is plausibly the same strained workspace/upstream provider refusing under load.Where to look
[ThreadExec] ERRORcatch site inMeshWeaver.AI.AgentChatClient— wrap theCompleteChatStreamingAsyncenumeration in atry/catch (ClientResultException ex)and mapex.Status: 429 → bounded retry with exponential backoff (honourRetry-Afterwhen present; this sample had none), 402 → reducemax_tokens/ surface quota exhaustion, 404/5xx → domain error message. The chain to fix above isOpenAI.ClientPipelineExtensions.ProcessMessageAsync→OpenAI.Chat.ChatClient.CompleteChatAsync→Microsoft.Extensions.AI.OpenAIChatClient.FromOpenAIStreamingChatCompletionAsync.Known-related / likely duplicate
This is the same unhandled-escape defect already filed as Systemorph/MeshWeaver#5932 (HTTP 402, parent incident
Admin/_LogIncident/828094ade6813b76, still active) and #2233 before it; the 429 case was explicitly called out there as the same catch site with a different remedy. A human can likely close this as a duplicate of #5932 once the shared status-mapping handler lands — but the 429-specific retry-with-backoff behaviour is not covered by #5932'smax_tokensfix, so it needs its own closing condition.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-65b5c866cd-6bkwhRecent log lines
systemorph-com commentedon Oct 1, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
9f4ad2732d4debf3(MeshWeaver.AI.AgentChatClient), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.0b9d6eb8fda3298a9f4ad2732d4debf3MeshWeaver.AI.AgentChatClientSystem.ClientModel.ClientResultExceptionOpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)HTTP {n} (: ) Provider returned errormemexmemex-portal-deployment-65b5c866cd-6bkwhAdmin/_LogIncident/0b9d6eb8fda3298aWhat triage wrote about this fingerprint
What is failing
During an agent thread-execution round, the OpenAI-backed chat completion call was rejected by the provider with HTTP 429 (rate limit). The
ClientResultExceptionpropagated up through the OpenAI streaming pipeline intoAgentChatClient, which failed the round and logged it at Error. A single transient rate-limit response from the model provider takes down a whole chat round.Probable cause
The OpenAI provider integration has no retry or backoff for 429 responses, so the first rate-limit hit kills the round instead of waiting and re-issuing the request. High confidence that the failure itself is a provider-side rate limit (status 429 with an empty error body); medium confidence on the fix location, since the MeshWeaver frames of the stack are truncated — the entry point into our code is the
MeshWeaver.AI.OpenAIstreaming-completion adapter.A secondary observation: the 429 response carried no reason phrase and no error body ("HTTP 429 (: ) — Provider returned error"), so whatever the provider sent is not being surfaced. Worth checking whether the pipeline should extract and log the response body/
Retry-Afterheader.Impact
Single occurrence on one pod so far — one agent round (a PR-review triage thread) failed and needs reactivation. A nuisance today, but the same code path means a sustained rate limit (e.g. an org-wide quota exhaustion) would fail every round on every thread with no self-recovery.
Where to look
MeshWeaver.AI.OpenAIassembly — the streaming completion adapter aroundOpenAIChatClient.FromOpenAIStreamingChatCompletionAsync.MeshWeaver.AI.AgentChatClient([ThreadExec] ERRORlog site): decide whether a 429 from the provider should fail the round outright or be retried with backoff (honouringRetry-Afterwhere present).Related
MeshWeaver.AI.AgentChatClientalso logs the separate, already-trackedHISTORY_LOAD_FAILED-on-shutdown defect (Systemorph/MeshWeaver.Plugins#2625) — same log category, different failure mode; this ticket is not a duplicate of it. No existing issue covers provider 429/rate-limit handling.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-65b5c866cd-6bkwhRecent log lines
systemorph-com commentedon Oct 1, 2026
Still occurring: 11 more since the last update (45 total).
memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-85469d6f4b-lrxh7,memex-portal-deployment-85469d6f4b-2gwtz,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-65b5c866cd-mwb67,memex-portal-deployment-65b5c866cd-szgkhRecent log lines
Re-addressed by the current identity function: this incident inherited
11973e8dfa3d0711(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 1, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5Recent log lines
systemorph-com commentedon Oct 1, 2026
Still occurring: 3 more since the last update (5 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86Recent log lines
systemorph-com commentedon Oct 1, 2026
Still occurring: 2 more since the last update (7 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsqRecent log lines
systemorph-com commentedon Oct 2, 2026
Still occurring: 1 more since the last update (8 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqzRecent log lines
systemorph-com commentedon Oct 2, 2026
Still occurring: 1 more since the last update (9 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-59f87cfd98-br5b9Recent log lines
systemorph-com commentedon Oct 3, 2026
Still occurring: 1 more since the last update (10 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmkRecent log lines
systemorph-com commentedon Oct 3, 2026
Still occurring: 3 more since the last update (13 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-67b688cf6f-bndxb,memex-portal-deployment-7dd57d4d64-c5rv8Recent log lines
systemorph-com commentedon Oct 3, 2026
Still occurring: 1 more since the last update (14 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-67b688cf6f-bndxb,memex-portal-deployment-7dd57d4d64-c5rv8,memex-portal-deployment-6bd968f77b-6745fRecent log lines
systemorph-com commentedon Oct 4, 2026
Still occurring: 3 more since the last update (17 total).
memex-portal-deployment-747567ff6d-zsvvw,memex-portal-deployment-6f5fcb56d6-8d4d5,memex-portal-deployment-7bcfbfcc8-6mkhf,memex-portal-deployment-7975d75fb-zk5l2,memex-portal-deployment-6b555484b4-qjm86,memex-portal-deployment-6f8fcf49d6-2nmsq,memex-portal-deployment-6f6c5b9978-mqvqz,memex-portal-deployment-59f87cfd98-br5b9,memex-portal-deployment-5d99755fbf-mkpmk,memex-portal-deployment-67b688cf6f-bndxb,memex-portal-deployment-7dd57d4d64-c5rv8,memex-portal-deployment-6bd968f77b-6745f,memex-portal-deployment-57d6d7f9cc-tslft,memex-portal-deployment-7b74d6bfb7-gwdkb,memex-portal-deployment-57f96dbd4d-4ww94Recent log lines
49 remaining items
systemorph-com commentedon Oct 12, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (52 total).
memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-85469d6f4b-lrxh7,memex-portal-deployment-85469d6f4b-2gwtz,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-65b5c866cd-mwb67,memex-portal-deployment-65b5c866cd-szgkh,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-7779877d5d-z2trfRecent log lines
Re-addressed by the current identity function: this incident inherited
11973e8dfa3d0711(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 12, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (52 total).
memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-85469d6f4b-lrxh7,memex-portal-deployment-85469d6f4b-2gwtz,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-65b5c866cd-mwb67,memex-portal-deployment-65b5c866cd-szgkh,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-7779877d5d-z2trfRecent log lines
Re-addressed by the current identity function: this incident inherited
11973e8dfa3d0711(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Oct 12, 2026
Re-closing: the 2026-10-12 15:43Z reopen is a DIFFERENT fault that shares this fingerprint
What reopened it. The bot's own log line for the new occurrence (15:43:18Z, pod
…-7779877d5d-z2trf) is not the PII block. It is an HTTP 402:This request requires more credits, or fewer max_tokens. You requested up to 131072 tokens, but can only afford 96267 … adjust the key's daily limit. The two 10-05 lines quoted in the same comment are 402s as well.Why it folded here. The fingerprint (
367de820103e2273) is categoryMeshWeaver.AI.AgentChatClient+ClientResultException+ top frameClientPipelineExtensions.ProcessMessageAsync. That covers every provider refusal on that path, whatever the status code. The PII 403 and the credit 402 cannot be told apart by it.The PII block has not recurred.
Ops/Actions/w5932-memexcloud-rcl-24h-20261012:{namespace="memex-cloud"} |~ "(?i)(redaction_context_lost)|(PII detected)", last 24 h, 0 lines, not cut.Ops/Actions/w5932-memex-rcl-402-24h-20261012on memex, same pattern pluscan only affordas the positive control: 100 lines, cut. Every landed entry I read is the 402 /[ProviderBudget] PROVIDER_KEY_FAILING holder=Provider/OpenRouterside. So the instrument reads that namespace, and the window is full of the credit refusal. Because that read is cut, it is not by itself a zero for the PII pattern on memex.redaction_context_loston record is still 2026-10-09 11:00:03Z (closing comment above).The 402 is live right now. Two probe rounds I started at 16:32Z, one on memex.systemorph.com and one on the control instance, both ended
the provider refused it for lack of credit or budget (HTTP 402)onz-ai/glm-5.3. That is an account/budget act for the owner of the OpenRouter key (raise the key's daily limit or top up). It is not this issue. Separately, every request asks formax_tokens131072, so a round fails while the key still has ~96k tokens of budget. Whether to cap that is a code question for its own ticket.Closing again on the reading above. If the bot reopens this on another 402, the fingerprint split is the fix, not this issue.
systemorph-com commentedon Oct 12, 2026
Reopened — this fault is happening again.
Still occurring: 184 more since the last update (236 total).
memex-portal-deployment-7fc5f4874b-tl6zn,memex-portal-deployment-8d98c8fc6-f674p,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-85469d6f4b-lrxh7,memex-portal-deployment-85469d6f4b-2gwtz,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-65b5c866cd-mwb67,memex-portal-deployment-65b5c866cd-szgkh,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-7779877d5d-z2trf,memex-portal-deployment-54486c96c4-msxhn,memex-portal-deployment-7779877d5d-sf5z9,memex-portal-deployment-7779877d5d-sr5w9,memex-portal-deployment-7945b8d984-8rszg,memex-portal-deployment-7945b8d984-glj9m,memex-portal-deployment-7945b8d984-6g8gm,memex-portal-deployment-6465785d4d-wjx78,memex-portal-deployment-6465785d4d-5h828,memex-portal-deployment-5b9996d768-hn2z5Recent log lines
Re-addressed by the current identity function: this incident inherited
11973e8dfa3d0711(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Oct 13, 2026
Re-closing. The 2026-10-12 21:53Z reopen is the HTTP 402 credit refusal again, not the PII block.
ClientResultException: HTTP 402 … This request requiresmore credits. That is the same shape the 2026-10-12 16:33Z comment separated out.Ops/Actions/wmlb-6223-memex-closed-24hon control):{namespace="memex"} |~ "(?i)(is CLOSED until)|(redaction_context_lost)"over 2026-10-12 14:58Z to 10-13 14:58Z returned 185 lines, not cut. All 185 are… is CLOSED until … (provider InsufficientCredits (402)). 0 areredaction_context_lost. Unlike yesterday's memex read, this one is not cut, so it is a clean zero for the PII pattern on memex over 24 h. The lastredaction_context_loston record is still 2026-10-09 11:00:03Z.AgentChatClientClientResultExceptioninto one fingerprint, whatever the status (MeshWeaver.Plugins#3499). MeshWeaver.Plugins#3502 (merged 2026-10-13 12:13Z) keys a provider refusal on its error class (HTTP status plus provider code). Once that rolls, a 402 files under its own identity instead of reopening this issue.max_tokens(MeshWeaver.Plugins#3498). It is not this issue.systemorph-com commentedon Oct 16, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (209 total).
memex-portal-deployment-6ddc8c645b-wlf5j,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-l7dp7,memex-portal-deployment-67c67b456d-s7nt9,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-5bbc8b6d-7rpnz,memex-portal-deployment-848f664fd5-zzqll,memex-portal-deployment-848f664fd5-zqfrh,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-5dcbbd8f4b-jqqxx,memex-portal-deployment-5dcbbd8f4b-6kzlg,memex-portal-deployment-5dcbbd8f4b-w5g96,memex-portal-deployment-7c4764b686-tbr7f,memex-portal-deployment-58475df96-ggzbx,memex-portal-deployment-7c4764b686-wm4t6,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-f86c77f48-g6pwj,memex-portal-deployment-f86c77f48-45tjv,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-65d864bddf-wtmvs,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-67dbbcf9b8-fnhhx,memex-portal-deployment-67dbbcf9b8-dv76t,memex-portal-deployment-5d74648df-ht6qx,memex-portal-deployment-8669548545-hnbx7🚨 This incident covers 2 distinct message shapes, so its own Last seen is the newest of ALL of them. Read the row for the shape this issue is about — a shape whose last sighting predates a fix is not recurring, whatever the counter says.
a3b6af2655bfc231— HTTP {n} (: ) Provider returned error0b9d6eb8fda3298a— HTTP {n} (: ) Provider returned errorRecent log lines
Re-addressed by the current identity function: this incident inherited
0b9d6eb8fda3298a(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph-com commentedon Oct 16, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (209 total).
memex-portal-deployment-6ddc8c645b-wlf5j,memex-portal-deployment-65b5c866cd-6bkwh,memex-portal-deployment-75f7dd4cfb-mk985,memex-portal-deployment-67c67b456d-l7dp7,memex-portal-deployment-67c67b456d-s7nt9,memex-portal-deployment-67c67b456d-255rq,memex-portal-deployment-5bbc8b6d-7rpnz,memex-portal-deployment-848f664fd5-zzqll,memex-portal-deployment-848f664fd5-zqfrh,memex-portal-deployment-848f664fd5-jgkbf,memex-portal-deployment-5dcbbd8f4b-jqqxx,memex-portal-deployment-5dcbbd8f4b-6kzlg,memex-portal-deployment-5dcbbd8f4b-w5g96,memex-portal-deployment-7c4764b686-tbr7f,memex-portal-deployment-58475df96-ggzbx,memex-portal-deployment-7c4764b686-wm4t6,memex-portal-deployment-58475df96-nprkt,memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-6cfcf78898-kss8q,memex-portal-deployment-6cfcf78898-lsklh,memex-portal-deployment-f86c77f48-g6pwj,memex-portal-deployment-f86c77f48-45tjv,memex-portal-deployment-f86c77f48-5fmjr,memex-portal-deployment-65d864bddf-v5s6s,memex-portal-deployment-65d864bddf-wtmvs,memex-portal-deployment-6bcdbb54fc-g9gh4,memex-portal-deployment-67dbbcf9b8-fnhhx,memex-portal-deployment-67dbbcf9b8-dv76t,memex-portal-deployment-5d74648df-ht6qx,memex-portal-deployment-8669548545-hnbx7🚨 This incident covers 2 distinct message shapes, so its own Last seen is the newest of ALL of them. Read the row for the shape this issue is about — a shape whose last sighting predates a fix is not recurring, whatever the counter says.
a3b6af2655bfc231— HTTP {n} (: ) Provider returned error0b9d6eb8fda3298a— HTTP {n} (: ) Provider returned errorRecent log lines
Re-addressed by the current identity function: this incident inherited
0b9d6eb8fda3298a(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Oct 16, 2026
Closing: the PII-redaction block in the title is gone.
Logsreadspr16-5932-redaction-memex-72h-20261016and…-memexcloud-72h-…, query(redaction_context_lost)|(PII detected), 2026-10-13 13:12Z to 10-16 13:12Z: 0 lines on each portal, not cut. The last occurrence was 2026-10-09 11:00Z.What kept reopening it is a different fault:
HTTP 400 (: ) Provider returned errorin agent rounds, incidentAdmin/_LogIncident/a3b6af2655bfc231, which reached this issue through an earlier fold. It now has its own ticket, Systemorph/MeshWeaver.Plugins#3759, and the incident points there (repository,issueNumber,issueUrl,occurrencesAtLastComment: 211).