Repository navigation
[workflow][infra] Bind and run both break cluster-wide from ~14:15 SGT; identical definition passed 12 minutes earlier (admission SOURCE_CREDENTIAL_REQUIRED + code_execute 502 + Orleans forwarding failure) #3290
Description
Workflow bind and run both break cluster-wide from ~14:15 SGT; identical definition passed 12 minutes earlier
Summary
Between 14:03 and 14:20 SGT on 2026-08-07 (06:03–06:20 UTC), our scope went from fully working to unable to bind or run any workflow. Nothing changed on our side in that window — we can prove this because the identical workflow definition that passed admission at 14:03 and completed all 14 steps at 14:15 is rejected at admission when resubmitted byte-for-byte at 14:20, and still rejected at 14:34.
Four distinct symptoms appeared together. All are infrastructure-level; none reproduce against NyxID or the downstream service directly.
Timeline (SGT, UTC+8)
| Time | Event |
|---|---|
| 14:03 | workflows:explicit-request-preview returns 6 call sites; bind succeeds |
| 14:04 | Run rejected with workflow run is already active while every run in the list reads failed; step count regresses 9/10 → 6/10 |
| 14:15 | Same member runs 14/14, completed |
| 14:20 | Identical definition rejected: NYXID_ADMISSION_SOURCE_CREDENTIAL_REQUIRED |
| 14:20 | Runtime code_execute fails: NYXID_PROXY_HTTP_502 |
| 14:30 | Second workflow (27 steps) fails at step 1/27: Forwarding failed: tried to forward message Request [<internal-silo-address> runtimeactor/sco… |
| 14:34 | Admission still rejected (4th reproduction) |
Symptom 1 — admission rejects a definition it accepted 17 minutes earlier
POST /api/scopes/{scopeId}/workflows:explicit-request-preview
400 INVALID_USER_WORKFLOW_REQUEST
External workflow capability admission failed:
NYXID_ADMISSION_SOURCE_CREDENTIAL_REQUIRED: A source-readable caller NyxID credential is required.
Reproduced 4 times (14:20, 14:21, 14:22, 14:34) with the byte-identical request body that succeeded at 14:03.
Isolation probes — we first assumed this was specific to a write call site we were adding, and tried to narrow it. All four returned the same error:
| Probe | Result |
|---|---|
POST /open-apis/im/v1/messages (the new write) |
SOURCE_CREDENTIAL_REQUIRED |
same call site changed to GET |
SOURCE_CREDENTIAL_REQUIRED |
| same call site pointed at a different path | SOURCE_CREDENTIAL_REQUIRED |
| same call site degraded to the exact shape of a read step that had just passed | SOURCE_CREDENTIAL_REQUIRED |
That assumption was wrong. The control settled it: removing the new call site entirely and resubmitting the definition that ran 14/14 minutes earlier is also rejected. So this is not about any call site we authored.
Both capability selectors are affected — capability.nyxid_operation and capability.nyxid_request behave identically.
Symptom 2 — runtime code_execute returns 502 while the sandbox is healthy
An already-bound workflow (bound 14:03, ran 14/14 at 14:15) now fails at step 8:
tool 'code_execute' execution failed: NYXID_PROXY_HTTP_502: The service request failed.
Calling the same sandbox service directly through NyxID at the same time succeeds:
{"success":true,"output":{"stdout":"ok","stderr":"","exit_code":0,"execution_time_ms":1089}}So the sandbox is up and the NyxID proxy path to it is healthy. The break is on the aevatar → sandbox leg.
Symptom 3 — Orleans message forwarding failure on a second workflow
A different member (27-step workflow, bound 2026-08-06) fails at step 1 of 27, before any of our code runs:
step_executor_failed: step 'readctx' (tool_call) failed during executor:
Forwarding failed: tried to forward message Request [<internal-silo-address> runtimeactor/sco
The message is truncated by the platform at ~100 characters, so we cannot supply more of it. The internal silo address is redacted here and can be provided privately.
Symptom 4 — run-actor state inconsistency
At 14:04, new runs were refused with workflow run is already active while the runs list showed every run in failed state — no active run existed. In the same window, the observatory step count for one run went backwards, 9/10 → 6/10, across successive polls of the same runId.
We initially logged this as an isolated anomaly and moved on when a retry succeeded at 14:15. In hindsight it looks like the earliest symptom of the same problem: a grain reactivating on a different silo with stale state would produce exactly this.
What is healthy (controls run at 14:34, same session, same token)
- chrono-sandbox direct via NyxID — 200,
stdout: "ok", 1130 ms - Lark bot service direct via NyxID — reaches upstream, returns the upstream's own scope error
- Caller identity —
whoamiresolves; token valid for another ~4.5 h - aevatar read endpoints — scope, members, bindings, observatory runs all respond normally
So: NyxID is up, the downstream services are up, our credential is valid, and aevatar's read plane works. Admission, the runtime tool leg, and actor dispatch are what changed.
Our reading
One hypothesis covers all four symptoms: aevatar lost the ability to read or use the forwarded caller NyxID credential, alongside silo churn. The aevatar service is configured Auth: none and relies entirely on the forwarded caller token — admission needs it to resolve capabilities (symptom 1), and the runtime needs it as the bearer to the sandbox (symptom 2). Symptoms 3 and 4 point at grain relocation happening at the same time.
We are not attached to this explanation; the timeline and the controls are the durable part.
Impact
Both of our production-migration workflows are fully blocked — one that had just reached end-to-end success for the first time, and one that cannot get past its first step. Our production automation still runs on the previous system, so there is no business outage; the migration is what is stalled.
Environment
- Scope, member, workflow and run IDs: available privately on request
- Reproduced continuously from 14:20 to 14:34 SGT on 2026-08-07
- All requests issued through the NyxID proxy with a caller token valid until 19:00 SGT
Activity
jianwei-su commentedon Aug 7, 2026
One-line summary: P2 first ran 14/14 at 14:09 today with numbers matching our current production system exactly, but since 14:15 the identical definition fails admission (
NYXID_ADMISSION_SOURCE_CREDENTIAL_REQUIRED), runtimecode_executereturns 502, and P1 dies on step 1 withForwarding failed(0/27) — while the sandbox answers 200 directly and both the NyxID proxy and our token are fine, so both migration lines are down. Please start with cluster silo stability.Additional evidence gathered after filing.
Three consecutive runs on the same member, same binding, same definition, inside nine minutes — we changed nothing between them:
workflow run is already active(no run was actually active)code_execute→NYXID_PROXY_HTTP_502The middle row is the one to look at: that binding is healthy and the definition is correct. Nine minutes later the same thing fails on a tool call, and the step count in the failing run is smaller than the successful run's total, which is consistent with the state regressions described in the issue body.
Admission has now been reproduced five times (14:20, 14:21, 14:22, 14:34, 14:42) with the byte-identical request body that succeeded at 14:03.
Controls re-run at 14:42, unchanged: sandbox direct through NyxID returns 200 (
stdout: ok, 1130 ms); the NyxID proxy is healthy; our caller token is valid for several more hours andwhoamiresolves.Impact on us: P2's migration is functionally complete and independently reconciled — the only remaining step is one re-bind, which requires
explicit-request-previewto work. P1 cannot be assessed at all, because the forwarding failure happens before any of our workflow code executes.Scope, member, run and revision IDs, plus the full internal silo address in the truncated Orleans message, are available privately on request.
eanz17 commentedon Aug 7, 2026
补充一条与本 issue 中
code_execute 502相关的归因更正，来自对部署版 chrono-sandbox 源码的直接核对。Aevatar 此前把 502 一律报成「容量不足」
NyxIdManagedCodexChronoTransport.ProxyFailure仅按 HTTP 状态分类，429 or 502 or 503 => CapacityUnavailable。而验收面看到的稳定码由CodexExecutionFailureKind派生，因此所有 502 都显示为codex_execution_capacity_unavailable，与真实故障阶段无关。部署版 chrono-sandbox 的真实语义（
feat/managed-codex-execution@1e8134d8）CODEX_CAPACITY_UNAVAILABLE只在容量许可被拒时产生，且走 429（src/codex/service.rs的too_many_requests）——不产生 502。create_codex_sandbox失败，携带CODEX_SANDBOX_CREATION_FAILED/CODEX_OPENSANDBOX_UNAVAILABLE/CODEX_OPENSANDBOX_TIMEOUT（src/services/sandbox/codex.rs:358,804,825）。即：看到 502 时，真实原因是沙箱创建 / OpenSandbox 侧故障，不是容量不足。 生产观测到的「约 11.1 秒后 502」也符合「尝试创建沙箱后失败」而非瞬时容量拒绝。
5f51f6d0已让非 2xx 保留 body 并提取 allowlisted 上游码，但该码此前只影响Failure.Code、不影响Kind，而验收稳定码由Kind派生，所以真实 stage 在验收侧不可见。已修复
0d900b66f（已推feature/integrate）：存在 allowlisted 上游码时由该码决定Kind—— sandbox creation / workspace preparation / OpenSandbox unavailable →ProvisioningFailed；各类 timeout →TimedOut；capacity →CapacityUnavailable；无上游码时保持按状态回退。不新增 fallback、不放宽 identity/credential 契约、不改请求形态。另修正两条把该误标固化成契约的既有断言（
CODEX_SANDBOX_CREATION_FAILED与CODEX_WORKSPACE_PREPARATION_FAILED原本被断言为CapacityUnavailable）。验证：ChronoSandbox.Tests 222 passed（新增 6 例 theory）、AI.Tests 2480 passed、Integration.Tests 486 passed / 4 skipped、architecture + stability guards 通过。
对本 issue 的意义
如果本 issue 记录的
code_execute 502也经由 managed codex 通道归因过，那条「容量」结论需要重新审视——502 指向的是沙箱创建/OpenSandbox，与SOURCE_CREDENTIAL_REQUIRED和 Orleans forwarding failure 是否同源需要用新的 typed stage 重新观测。部署包含0d900b66f的镜像后，同类失败会给出可区分的ProvisioningFailed/TimedOut/CapacityUnavailable。🤖 Generated with Claude Code
jianwei-su commentedon Aug 11, 2026
Re-tested today, 2026-08-11, around 14:00 SGT. Two of the three problems in this issue are fixed. The third has changed shape twice and still blocks both workflows.
First, thank you for the 502 attribution correction — we had taken the "capacity" label at face value and reasoned from it, so knowing that a 502 points at sandbox creation / OpenSandbox rather than capacity is genuinely useful. We had no way to see that from the outside.
Fixed — confirmed by live test
Orleans message forwarding. The 27-step workflow that previously died at dispatch with
Forwarding failed: tried to forward message Request […]now reaches its first step and fails there instead. Dispatch itself is working.Admission.
workflows:explicit-request-previewaccepts the definition again and returns all 6 call sites; the subsequent bind was accepted. This is the same definition that was rejected five times on Aug 7.One thing worth flagging: the preview response shape changed. It used to return a
callSitesarray and now returnsitems. Our tooling keyed oncallSitesand read the new response as a failure until we looked at the raw body. If that was intentional, it is a breaking change for anyone parsing the response; if not, it may be worth a look.Still blocking — and it is a new error
Both workflows now fail on their first
code_executestep:"Not uniquely available" reads like an ambiguity rather than an outage — as if the router sees more than one candidate and cannot choose.
We have ruled out our side of that ambiguity. Our NyxID account has exactly one sandbox service, on a single endpoint, with no duplicate registration among our 37 connected services. We cannot enumerate what the scope sees on your side — the scope-level service and capability endpoints all return 404 to us — so if there is a duplicate candidate, it is in aevatar's own registry.
The error has changed twice in four days
NYXID_PROXY_HTTP_502code_execution_credential_unavailable: A source-readable NyxID credential is required for code execution.code_execution_route_unavailable: The code execution service is not uniquely available.The middle row may be worth separating out. That run was fired by the scheduler, so there was no interactive caller whose credential could be forwarded. If code execution resolves its credential from the forwarded caller token, scheduled runs have no such token by construction, and the weekly schedule would keep failing even after the routing problem is fixed. We would rather raise that now than discover it later.
Control, run in the same window
Calling the sandbox directly through the NyxID proxy works normally:
{"success": true, "output": {"stdout": "42", "exit_code": 0, "execution_time_ms": 1167}}So the sandbox is healthy and the NyxID path to it is healthy. What cannot resolve is aevatar's route to it.
Where that leaves us
The migration itself is finished and independently reconciled; both workflows are waiting purely on code execution. Happy to re-run either one on request once a fix lands — we can turn a verification around quickly, and the definitions have not changed.
Scope, member, run and revision IDs available privately on request.
nwnwnw413 commentedon Aug 11, 2026
Independent reproduction of the third symptom, on a different scope. We hit the identical error today with no contact between our two migrations, which should rule out anything account-specific.
Scope
2c5c9b72-5b63-4c57-89bc-5d5492efebc3. All times UTC, 2026-08-11.Smallest possible repro
A one-step workflow with no external call at all — pure JS arithmetic:
Workflow
probe_code_exec, revisionrev-e3cf718d-63e3-4f5d-82c9-e6fd03c3b396, still bound in our scope. Four consecutive runs, identical result, 670–3300ms each:def22b6e631946cc908b4b2185b3e04e·c710a9f6e75b4f97939f3ffe498090f4·d41fb857fb5a4103bbbcc5c260c03f13·a6e8b5a0aeb54736a57950dd3c4e7946Full response body:
{"success":false,"error":"code_execution_route_unavailable", "code":"code_execution_route_unavailable", "message":"The code execution service is not uniquely available.", "diagnostic_id":null}Same failure in our real workflow:
community_radarrevrev-13c497d0-65c6-4d74-850c-a5539bf22f2e, runaa5ea0e18f4540c5b591f60924e40a81(started06:43:09.553). Stepparse_group1ran06:44:38.926 → 06:44:42.255.read_seenplus fournyxid_proxyGET fetches in that same run all succeeded first — so the run credential is fine fornyxid_proxyand fails only forcode_execute.Same conclusion on the ambiguity, reached independently
We also have exactly one code-execution-capable service —
chrono-sandbox=2716aa5a-5ece-4e8f-a238-6e39e8b03020,active, no duplicate among our connected services. So there is no second candidate on our side either.Given that, "not uniquely available" reading as ambiguity may be worth questioning: with exactly one candidate visible to each of two unrelated accounts, resolving to zero fits the evidence at least as well as resolving to many. That is an inference from the outside, not a claim about your resolver.
Our
chrono-sandboxshowsAuth: none / Credential: noneandaccess: members, so we cannot alter its delegation configuration ourselves.Control, same window
Sandbox healthy, NyxID path healthy, aevatar's route to it is what fails — matching what @jianwei-su reported above.
Cross-reference
We filed #3284 for the earlier
NYXID_PROXY_UNAUTHORIZEDform of this on Aug 7; it was fixed by imagecb04697d, and the failure then became this one. Our re-test is in #3284 (comment). Given the error progressionNYXID_PROXY_UNAUTHORIZED→code_execution_route_unavailableon our side and502→code_execution_credential_unavailable→code_execution_route_unavailableon @jianwei-su's, these look like the same underlying resolver work in progress, plausibly #3328's delegation-only migration. If so, it would help to know which step of #3328 makes it safe to re-test.We can re-run the one-line probe as a canary in under a minute whenever it is useful.
cc @eanzhao
eanz17 commentedon Aug 12, 2026
Fixes for the remaining
code_executeblocker are now deployed to mainnet in imageaelfdevops/aevatar-console-backend:fd30ce36.Included fixes:
620cb2674- select and freeze the exact catalog-backedcode_executeUserService during v5 admission, including personal routes and organization routes allowed for the caller.e15008cda- authorize scheduled code execution from the admitted durable route.783a4d3fa- use the correct delegation credential kind for scheduled workflow dispatch.f1644e6bb- preserve the admitted delegation credential through scheduled execution.Production verification on the deployed image:
code_executecanary through the authenticated NyxID proxy path.cbca4e5f2e954860b9cde203a5bb6375.chrono-sandboxUserService ID.success=true,exit_code=0, stdout{"ok":true,"sum":2}, execution time1141 ms.lastSuccess=true,stateVersion=17.code_execution_route_*orcode_execution_credential_unavailablerejection.Verification at the deployed revision:
NyxIdCodeExecuteToolTestsandNyxIdCodeExecutionWorkflowCapabilitySourceTests: 37/37 passed.tools/ci/test_stability_guards.sh: passed.Please re-run the existing interactive one-line probe on both affected accounts. For scheduled execution, please re-preview and re-bind on the current image before testing the schedule: delegation-only scheduled execution requires the exact v5 admission proof produced by the new admission path.
I am not claiming a live scheduled-run success yet. My schedule canary was rejected with HTTP 400 before any schedule or run was created because that temporary workflow had interactive rather than durable admission. The scheduled path is covered by the deployed implementation and tests, but it still needs reporter-side validation against a correctly re-bound durable workflow.
I am keeping this issue open until both affected routes are confirmed, especially the organization-inherited route. The preview response currently intentionally serializes the collection as
items; the earliercallSitescompatibility concern was not changed as part of this fix.jianwei-su commentedon Aug 12, 2026
Re-ran on the deployed image as asked. The interactive path still fails for us, and the reason looks specific: our sandbox service is not catalog-backed.
All times SGT (UTC+8), 2026-08-12.
What we did, in order
code_executewithcode_execution_route_unavailableINVALID_SCOPE_SERVICE_INVOKE_REQUEST: Failed to initialize Orleans actor scope-workflow:…explicit-request-previewThe re-bind rejection:
So the old binding is correctly invalidated by the new image, re-preview works, and the new admission path rejects us at bind time.
Why we think it is the catalog-backed requirement
Your note says admission now selects and freezes the exact catalog-backed
code_executeUserService. We checked our side:chrono-sandbox,active,Auth: none,Credential: none, endpointhttps://sandbox.chrono-ai.fun/. No duplicates among our 37 connected services.chrono-sandboxis not in the NyxID catalog. The catalog lists 31 services;chrono-llmis there,chrono-sandboxis not. Ours was added as a custom HTTP service, not from a catalog entry.If admission now requires the selected code-execution UserService to resolve to a catalog entry, then an account whose sandbox was added as a custom service has no way to satisfy it — which would match the
contract_driftclassification and the "Restore platform code route" remediation, neither of which we can act on from our side.That would also explain why your canary passed and ours does not: if your sandbox is catalog-backed and ours is not, the deployed fix is correct but does not cover this shape of account.
We are not certain this is the cause — it is the one difference we can see from outside. If you can tell from the admission proof which UserService was selected (or why none was), that would settle it quickly.
What would unblock us
Either of these, whichever matches the intended design:
chrono-sandboxto the NyxID catalog so our service can be catalog-backed, orOne more observation, possibly unrelated
The two access paths have swapped behaviour across the last day, which made diagnosis harder:
scopeResolved: truestudio/contextreturnsscopeId: null, scopeResolved: falseSame token, valid in both windows. We mention it because your canary ran "through the authenticated NyxID proxy path", which is currently the failing path for us — so the two of us may not be exercising the same route.
Happy to re-run anything on request; the workflow and bindings are unchanged and a retest takes a few minutes.
jianwei-su commentedon Aug 12, 2026
Following up on the catalog point, because it is the kind of claim that is easy to get wrong and I wanted to rule out the obvious false positive before you spend time on it.
The obvious objection, tested and ruled out
nyxid catalog listprints under the heading "Available Services" with a "How to Add" column. That phrasing suggests it might only list services the caller has not added yet — in which casechrono-sandboxwould be missing from the listing simply because we already have it, and my earlier conclusion would be wrong.It does not work that way. Services we have already added still appear in the catalog listing:
api-lark-botllm-openaiapi-larkchrono-sandboxSo the listing is not filtered by what the caller already has, and
chrono-sandbox's absence is real rather than a display artifact. The onlychrono-*entry in the catalog ischrono-llm.What our sandbox actually is
It was added as a custom HTTP service, not from a catalog entry. It is our only code-execution-capable service, with no duplicates among 37 connected services.
How this lines up with the deployed fix
Your change selects and freezes the exact catalog-backed
code_executeUserService during v5 admission. If our sandbox has no catalog entry behind it, there is nothing for that selection to resolve to — which would produce exactly thecontract_drift/CODE_EXECUTION_ROUTE_POLICY_MISMATCHwe now get at bind time, and the "Restore platform code route" remediation we cannot act on.It would also explain the asymmetry: your canary passes because your sandbox is catalog-backed; ours cannot, because it never was.
Still a hypothesis — we can only see our side. What would confirm or kill it in one step is the admission proof for our rejected bind: which UserService did the selector consider, and why was none accepted.
Two possible resolutions
chrono-sandboxto the catalog the waychrono-llmis, after which we re-add our service so it is catalog-backed.We have no preference; whichever matches the intended design. Option 1 needs someone with catalog write access, so it is not something we can do ourselves.
Ready to retest within minutes of either landing — the workflow and bindings are unchanged, and we have an automated check that re-binds and re-runs on its own.
eanz17 commentedon Aug 12, 2026
Thanks for the immediate retest. I correlated your bind request at
2026-08-12 11:15:11 SGTwith production logs.The server-side evidence is:
This rules out the non-catalog hypothesis. If the UserService had no
catalog_service_id, Aevatar would have reportedCODE_EXECUTION_ROUTE_MISSINGwithcanonicalCandidates=0. Aevatar is seeing exactly one catalog-backedchrono-sandboxroute for your caller, but that route's stored delegation policy is incompatible with the platform contract.The deployed resolver intentionally accepts both supported transition shapes:
forward_access_token=true,inject_delegation_token=true, scopeproxy:*orproxy:* sandbox:execute; orforward_access_token=false,inject_delegation_token=true, scopesandbox:executeorproxy:* sandbox:execute.Your route matches neither shape. This is consistent with a stale historical NyxID UserService identity-policy snapshot, not a missing catalog entry and not a reason to allow an arbitrary custom same-slug endpoint.
NyxID already has the generic, audited recovery path for this exact class of drift from NyxID#1378, merged in NyxID#1379:
That endpoint is platform-admin-only and force-aligns all UserService instances for the
chrono-sandboxcatalog entry to the catalog's current identity policy, with an audit record and affected count. Before running it, the operator must verify the current catalog policy; after it succeeds, please re-preview and re-bind once more.I have not claimed or performed that production mutation from a normal user credential. The issue remains open until the platform-admin resync is completed and both reporter accounts pass the interactive probe. The separate NyxID-proxy
401/scopeResolved=falseobservation is a different ingress/identity boundary and should not be inferred from this policy-mismatch result.jianwei-su commentedon Aug 12, 2026
Correction: I was wrong.
chrono-sandboxIS in the NyxID catalog. Please disregard my two previous comments on this point — the NyxID owner is right.What I got wrong
I ran
nyxid catalog list, did not findchrono-sandbox, and concluded it had no catalog entry. That command hides it by design:chrono-sandboxhasAuth Method: none, so it is a system service without auth and is omitted from the default listing. With the flag it is there:Worse, I then "ruled out" the obvious objection with a table showing
api-lark-bot,llm-openaiandapi-larkappearing in the listing despite being already added. That table was correct but irrelevant: those three all have auth methods, so the no-auth filter never applied to them. I tested the wrong hypothesis and reported the result as if it settled the question.So the catalog-backed theory is dead, and with it the two remediations I proposed. Nothing needs to be added to the catalog.
Where that leaves the actual problem
The failure itself is unchanged and still reproduces. On the deployed image, today:
INVALID_SCOPE_SERVICE_INVOKE_REQUEST: Failed to initialize Orleans actorCODE_EXECUTION_ROUTE_POLICY_MISMATCH,readiness.status: contract_driftOur side, restated without the bad inference:
chrono-sandbox,active, and it does have a catalog entry (Credential Mode: admin,Auth Method: none).success=true,exit_code=0, ~1.1s.I am not going to offer another theory. The one thing that would identify the cause directly is the admission proof for our rejected bind — which UserService the selector considered, and the specific reason each was rejected. If that can be surfaced, or if you can read it server-side for our scope, it should be a short step from there.
Apologies for the detour, and for stating a hypothesis with more confidence than the evidence supported.
jianwei-su commentedon Aug 12, 2026
First: my error, and why I made it
I reported that
chrono-sandboxhad no NyxID catalog entry. That was false. The NyxID owner is correct — it is in the catalog,Active,No auth required,Direct,Auto-connected. Anyone who acted on my comment was sent in the wrong direction, and I am sorry for the time that cost.How the error happened, precisely:
nyxid catalog list, did not seechrono-sandbox, and formed the hypothesis that it had no catalog entry.nyxid catalog list --help. It has a flag:--all — Include all active services (including system services without auth).chrono-sandboxhasAuth Method: none, so the default listing hides it by design. Default shows 31 entries;--allshows 36 and includes it.nyxid catalog show chrono-sandboxreturns a valid entry directly.api-lark-bot,llm-openaiandapi-larkappearing despite being added.Step 3 is the part worth naming. That table was accurate but tested the wrong thing: all three of those services have auth methods, so the no-auth filter never applied to them. I checked a hypothesis I had invented instead of checking the tool's own contract, got a confirming result, and reported with more confidence because of it. A false check is worse than no check — it converted a guess into a claim.
The catalog theory is dead. Nothing needs to be added to the catalog, and the two remediations I proposed should be discarded.
Second: what is actually failing now
Facts only, all measured today on the deployed image. No hypothesis attached.
chrono-sandboxin NyxID catalogsuccess=true,exit_code=0, ~1.1 sexplicit-request-preview, our 15-step workflowexplicit-request-preview, a one-stepcode_executeworkflowcode_executeworkflow, different member, zero confirmationsThe bind failure:
The last row is the one that matters. A workflow with one step, no external request, no confirmations, bound to a different member fails the same way. So this is not about our workflow definition, not about capability admission, and not about code-execution routing — the binding pipeline fails while initializing its own actor.
One more observation, offered without interpretation: within roughly one hour the same bind request returned two different errors —
CODE_EXECUTION_ROUTE_POLICY_MISMATCH(admission) at 11:15, thenINVALID_STUDIO_MEMBER_BINDING(actor init) from about 11:50 onward. Same payload both times. The second failure occurs before admission is reached.I am deliberately not proposing a cause. I have been wrong three times on this issue by reaching for an explanation early — the data structure difference behind the FIN-02 "regression", a transient token state I called a platform change, and the catalog claim above. The table is what I can stand behind.
Third: who this needs
aevatar side. The failing component is
studio.member-binding-runactor initialization, which is entirely server-side. Nothing in the workflow YAML, the NyxID catalog, or our service configuration can influence it — the one-step probe proves that.Nothing is needed from NyxID: the catalog entry exists and the sandbox responds correctly when called directly.
What would move this fastest is server-side logs for any of our failed
bind-<id>runs — the actor fails to initialize before returning anything we can see.We can retest within minutes; the workflows and members are unchanged, and we have an automated check that re-binds and re-runs on its own schedule.
jianwei-su commentedon Aug 12, 2026
The binding write returns
acceptedand then silently does not take effect. This is the current blocker, and it is a different class of problem from the earlier ones — a write that reports success while changing nothing.Reproduction, measured just now
The API accepts today's revision, returns success, and thirty seconds later the binding still points at yesterday's revision. No error is surfaced anywhere in that path.
invokethen fails in a way that confirms it:It looks for yesterday's revision — because the binding was never updated — and that definition actor was invalidated by the new image. So the two symptoms are one: the binding write is not persisted, and the stale revision it still points at no longer exists.
bindingRunId: bind-a525ad8151484aea95d361e33ab433d7should let you find this exact run server-side. Earlier ones from today:bind-fdd9f45ba7a54ae3bc71f78d33dddec7,bind-a3538688bad2471aa00e7f0209aa3ac7.Why we are flagging this one specifically
dispatch_acceptedappears to mean only that the request was queued. Whatever happens after that fails, and nothing reports the failure to the caller. From outside, a re-bind is indistinguishable from a successful one unless you separately read the binding back and compareboundAtandrevisionId— which is not something an API returning"status":"accepted"invites you to do.For our use this matters concretely. These workflows send payment-related cards. We keep a no-send variant bound at all times and swap to the sending variant only after verifying the numbers. If a swap reports
acceptedbut silently leaves the previous binding in place, the operator's mental model and the actual bound definition diverge with no signal. Today that would have failed safe, because the stale binding is the no-send one. The reverse ordering would not.We are not asking for a design change here — just that a binding write which does not persist should surface an error rather than
accepted.Today's sequence, for context
The blocker has moved down one layer at a time as fixes landed. All measured on our scope today:
explicit-request-preview)CODE_EXECUTION_ROUTE_POLICY_MISMATCHINVALID_STUDIO_MEMBER_BINDING(Orleans actor init)accepted, does not persistINVALID_SCOPE_SERVICE_INVOKE_REQUEST, stale revisiondraft-runexecutionThe last row is separate from the binding problem and may or may not share a cause — a one-step
code_executeworkflow and a template-only workflow both stream events without terminating. We are reporting it as an observation, not a diagnosis.What we need
bind-<id>runs above: what happens afterdispatch_accepted.acceptedresponse when the binding is not written.Nothing else is blocked on our side — admission passes, our definitions are unchanged, and we have an automated check that re-binds, verifies a no-send run reaches 14/14, and only then swaps to the sending variant. It has correctly refused to send at every gate so far.
nwnwnw413 commentedon Aug 12, 2026
New symptom on our scope as of today: no workflow can be invoked at all.
chat:streamreturns HTTP 500EXECUTION_FAILEDand no run is ever created. This is a step worse than yesterday, where runs started and then failed oncode_execute.Scope
2c5c9b72-5b63-4c57-89bc-5d5492efebc3. Measured2026-08-12T06:30–06:33Z.{"code":"EXECUTION_FAILED","message":"Workflow execution failed."} HTTP=500The control that makes this clean
probe_tolerance— a two-step workflow with nocode_executeat all (onenyxid_proxyGET plus anassign) — ran tocompleted / success=trueyesterday at2026-08-11T06:54:33Z. Same workflow, same revisionrev-c487e5a1-58f9-464f-9e5d-65f75fdebf84, definition untouched since. Today it returns HTTP 500 five times out of five.So this is not definition drift, and it is not specific to code execution.
GET /api/scopes/{scope}/servicesGET /api/workflow/observatory/runsGET /api/scopes/{scope}/workflows/community_radarPOST …/services/probe_tolerance/invoke/chat:streamPOST …/services/probe_code_exec/invoke/chat:streamPOST …/services/community_radar/invoke/chat:streamEvery service in the scope still reports
invokeReadinessStatus: Readywith a populateddefaultServingRevisionId, so readiness and the execution path disagree.The observatory confirms nothing is being created: the most recent run in our scope is still
eb44845b04eb417299bf3ed6e082ca1dfrom2026-08-11T06:54:33Z. Zero runs exist for 2026-08-12. The failure is upstream of run creation, which is why there is no run id to correlate — happy to provide anything else that would help.Separately — thanks for the
eligibleCandidatesbreakdownThat resolves the question I raised yesterday. I had suggested that "not uniquely available" reading as ambiguity was worth questioning, since two unrelated accounts each had exactly one candidate, and that resolving to zero fit the evidence at least as well. Your log line settles it in a way we could not have seen from outside:
One catalog-backed route, zero eligible after policy evaluation. That is a
PolicyMismatchsurfacing under a message that reads like an ambiguity. If the wording ever gets revisited,code_execution_route_ineligible(or anything naming the policy mismatch) would have saved both @jianwei-su and me from hunting for a duplicate registration that never existed. Minor, but it cost real time on two teams.Our one-line
code_executeprobe is still bound and takes under a minute to re-run whenever the route policy is fixed — assuming invocation itself comes back first.cc @eanzhao
eanz17 commentedon Aug 12, 2026
@jianwei-su Your "accepted-but-not-persisted" binding write is the same defect class we isolated this afternoon and filed as #3432, with the storage-side root cause in #3431. Short version, with what we changed on the infra just now:
Unified picture
--aof-size-limit 1gat ~03:00Z today. Past the cap, every commit is refused with errno 28, and truncation itself needs a commit — a permanent wedge. We restarted the pod ~04:00Z (recovery from checkpoint + committed AOF, no acknowledged-write loss).dispatch_accepted→ unchangedboundAt/revisionIdis exactly this signature, one subsystem over from oursave-and-bind→ stale canonical catalog. Identity-level readbacks pass; only a content/readback comparison catches it (your boundAt+revisionId check is the right instinct — we now hash the canonicalsource.workflowYamlafter every publish).bind-<id>traces you posted should settle it server-side.Infra changes just applied (aismart-mainnet)
BGSAVEtruncated the AOF 806M → 39M (it had refilled to 80% of the cap within ~2h of the restart — consistent with poisoned actors re-attempting their backlogs).garnet-aof-checkpointCronJob (BGSAVE every 30 min, modeled on the existinggarnet-backupinit container). The wedge only occurs when the cap is reached before a checkpoint lands, so this closes the recurrence window. The daily 03:00Z backup remains the durable snapshot.Still needed from the platform
accepted.For the poisoned actors themselves we're testing whether Orleans idle collection recycles them cleanly (a fresh activation rehydrates from the last committed state); if it does, an in-place republish lands and content readback goes green. Will report the result on #3432.
jianwei-su commentedon Aug 14, 2026
Re-tested on 2026-08-14. Yesterday this scope was healthy; today runs are accepted but never materialise — and it is not confined to
code_execute.Yesterday worked
08-13 09:26 SGT— our 14-step budget workflow rancompleted, 14/14 on this scope, the first clean end-to-end run since the incident. Thesendvariant then suspended correctly atawaiting_tool_approvalon its write call site, which is the designed gate and not a fault.So whatever landed after #3431/#3432 did work for us.
Today: three probes, one shared symptom
All measured within ~25 minutes on the same scope, same token.
code_executefailed 1/2—tool 'code_execute' execution failed: NYXID_PROXY_HTTP_524code_execute, invokerunning,0/0steps across 8 polls (~96 s) — nothing materialisedcode_executeat all, draft-runRUN_OBSERVATION_TIMEOUT— "Run was accepted but did not become observable before the deadline."Probe 3 is the one that changes the picture. That workflow's only tools are
nyxid_proxyreads — no code execution anywhere in it — and it still never becomes observable. Probes 2 and 3 share the same shape: the run is accepted, then no steps appear.Control, run in the same window: the sandbox answers directly through the NyxID proxy in ~1.09 s (
stdout: "42",exit_code: 0). So the sandbox is healthy; probe 1's 524 is on the aevatar→sandbox leg, and probes 2–3 do not depend on that leg at all.Why we are reporting it this way
Our earlier framing — "the aevatar→sandbox leg is broken" — was too narrow. Probe 3 has no such leg. What the three share is that a run is accepted and then produces no observable steps, which points at run materialisation rather than any one tool.
We are not proposing a cause. Probe 1 (524) may or may not be the same defect as probes 2–3; we have no evidence either way and are not going to guess. The useful fact is that a
code_execute-free workflow reproduces the non-materialisation.One incidental confirmation
Probe 3 was run twice. The first attempt, without a field projection on its four Bitable reads, failed at a 514,823-character context with
ERR Lua encountered an error— the same state-layer size problem as #3416. Addingfield_namesto the reads removed that error entirely, and the run then reachedRUN_OBSERVATION_TIMEOUTinstead. So the projection fix works and simply exposed the next layer.Also worth knowing: approval windows appear to expire
Yesterday's
sendrun is stillawaiting_tool_approval(14/15). Attempting to approve it today viaPOST /api/executions/{executionId}/resumereturns 404 — the execution is no longer resolvable, though the run still reports as awaiting approval in the observatory.If suspended runs expire after roughly a day while still displaying as pending, that is worth documenting: any workflow with a human approval gate can silently pass its approval window, and the operator sees a run that still looks actionable. We would rather know the intended TTL than discover it per-incident.
Happy to re-run any of these within minutes.
jianwei-su commentedon Aug 14, 2026
状态更新：批准接口已可用，此前的 404 不再复现
这条本来是要报「resume 路由全部 404」的，发之前重测了一遍，结论变了——先说结论：这一项已经好了。
此前观察（08-14 上午）
我方 P2 的 send 变体正确挂起于工具批准闸门后，按 run detail 里的
toolApproval四元组去调 resume，OpenAPI 里能找到的几条路由、多种参数组合全部 404。当时判断是路由缺失。现在（同日重测）
POST /api/executions/{executionId}/resume返回 400 而不是 404：{ "title": "One or more validation errors occurred.", "status": 400, "errors": { "RunId": ["The RunId field is required."], "StepId": ["The StepId field is required."] } }路由存在且可达，只是请求体还缺
RunId和StepId。这两个字段此前没有从toolApproval的四元组里体现出来——我们一直只带executionId/toolName/toolCallId/approvalRequestId，所以之前的尝试即使路由在也过不去。一个小建议：如果
toolApproval对象里能一并带上runId和stepId，调用方就不必再从别处拼装。目前这两个值要另行从 run 记录里取。挂起的 run 仍然存活，状态完好
08-14 01:37 UTC 那次运行至今仍在等待批准，observatory 里数据完整：
send步骤，outcome: waitingtoolApproval四元组齐全也就是说这个批准闸门本身一直工作正常——它按设计挂起、保持状态、等待人工决定，跨越了整场事故。
另一项也不复现了
此前报告过 observatory 响应里有非法
\uXXXX转义（约第 241,121 字符处），标准 JSON 解析器直接报错。本次同一个 run 的 observatory 响应 1.5 MB，可正常解析，未复现。不确定是已修复还是与具体内容相关，暂不下结论，先记录现象。
自我更正
我此前判断「待审批执行体一天后过期导致 404」是错的（当时用刚挂起几分钟的活跃 run 复现了同样的 404，已排除过期因素）。现在看，404 的成因也不是路由缺失——更可能是当时那条路径确实尚未就绪。这一项我先后给出过两个错误归因，都已在此更正。
jianwei-su commentedon Aug 14, 2026
更正上一条：批准接口没有修好，我判断错了
上一条我说「resume 路由已可用，只是缺
RunId/StepId」。这个结论是错的，请以本条为准。我把 400 当成了「路由通了」的证据，但没有把调用做完就下了结论。补上字段后再调，返回的是 404。
完整的实际行为
400 只是模型校验先于查找执行——它证明这条路由是注册过的，不证明批准链路可用。
真正的缺陷：observatory 与 resume 对同一个执行体不一致
同一时刻、同一个
executionId：GET /api/workflow/observatory/runs/{runId}send，outcome: waiting，toolApproval四元组齐全POST /api/executions/{executionId}/resumeobservatory 认为这个执行体存在且正在等待人工批准；resume 接口查不到它。两者取的是同一个 ID（直接从 observatory 的
toolApproval.executionId读出，未做任何加工）。run 状态完好：15 个步骤，14 succeeded + 1 waiting，等待的正是
send。工具批准闸门本身按设计工作——挂起、保持状态、跨越整场事故。唯一缺的是把批准送进去的通道。这个 run 仍然存活，可直接用于排查
我方不再对它做任何写操作，保持原状。需要
runId/executionId/approvalRequestId具体值的话可私下提供。关于我的判断
这是我在这个问题上第三次给出错误归因：先说「执行体过期导致 404」（错，刚挂起几分钟的活跃 run 同样 404），再说「路由缺失」，刚才又说「路由已修好」。
前两次的教训我写过——一次错误的检验比不检验更糟，它把猜测升级成断言。这次我又犯了同一类错：拿一个中间返回码当终局证据，没把动作做完就宣布结论。已在此更正。
上一条评论保留不删，作为这个更正的上下文。
jianwei-su commentedon Aug 17, 2026
08-17：单个
nyxid_proxy工具调用挂起 35 分钟不返回，同时段同 scope 的其他同类调用正常完成今天在同一 scope 下并行跑两个工作流（均为
submit=false预览、零副作用），观察到一个可复现的对照。现象：一个本该秒回的空查询挂了 35 分钟
工作流 A 的步骤
aa_p2（tool_call→nyxid_proxy，读取一张 Bitable 表的下一页）：这一步本来什么都不用做。它的前一步
aa_p1返回has_more: false——该表没有下一页。所以模板走的是「无更多分页」的兜底路径，生成了一个page_size=1的哑请求。也就是说：一个必然立即返回的空查询，挂了半小时以上。同一次运行中的同类步骤作为对照：
ca_p2读另一张表的第二页tool_call/nyxid_proxyaa_p1读同一张表的第一页tool_call/nyxid_proxyaa_p2哑请求（同一张表）tool_call/nyxid_proxy对照组：同时段同 scope 的另一个工作流，同一工具，正常完成
工作流 B 在同一时间窗内持续推进，其
nyxid_proxy调用都在正常完成并轮转：tool_call并发在飞所以不是
nyxid_proxy整体不可用，也不是凭据或路由问题——同一 scope、同一时刻、同一工具，B 的调用在正常返回，A 的那一个卡住不动。看起来是单个工具调用在派发或回收环节丢失，且没有超时兜底把它带出waiting状态。附带观察：全流程步骤耗时异常
即使是不调用任何外部服务的步骤也非常慢：
assigntransform（模板求值）两个工作流已完成步骤的耗时中位数分别是 25.7 秒和 185.6 秒。这类步骤在正常情况下应是毫秒级。
这可能与上面的挂起是同一根因（执行管道整体积压），也可能是两回事，我们无法从外部区分。
影响
我方的迁移验证长期受此影响：同一份定义两次运行时长相差一倍以上、给两倍超时反而跑得更少。此前我们一直把这些归因于工作流定义本身，现在看至少有一部分是执行侧的性能与挂起问题。
可提供
两个 run 的 runId、observatory 完整响应、以及各步骤的
requestedAtUtc/completedAtUtc/durationMs明细均可私下提供。挂起的那个 run 保持原状、不做任何写操作，可供直接排查。说明
此前我们在本 issue 报告的若干「工作流跑不到终态」，判断依据是 SSE 流未收到终态事件。现已确认流断开不等于运行结束——服务端运行仍在继续。这个判断口径是我们的错误，已改为轮询 observatory。本条中的所有时长均来自 observatory 的步骤时间戳，不依赖流。
jianwei-su commentedon Aug 17, 2026
更正：resume 路由一直存在，是我没找对；但它对 member 绑定的服务无效
此前我在本 issue 多次报告「工具批准无法送达，OpenAPI 声明的 resume 路由全部 404」。这个描述是错的，请以本条为准。
正确的路由
请求体：
{ "runId": "<完整 runId>", "stepId": "<挂起的步骤 id>", "approved": true, "commandId": "<新生成的 id>", "toolApproval": { "executionId": "...", "toolName": "...", "toolCallId": "...", "approvalRequestId": "..." } }关键是
/services/{serviceId}/。我此前尝试的是/api/executions/{executionId}/resume和/api/scopes/{s}/members/{m}/runs/{runId}:resume，都不是这一条。已实测通过：一个工作流服务在写调用点正确挂起后，用上述路由放行，
send_notify步骤succeeded（下游接口返回code: 0），整个 run 随后RUN_FINISHED。这是我们第一次完整跑通「运行 → 挂起于人工闸门 → 放行 → 执行写操作 → 正常结束」。仍然存在的问题：member 绑定的服务放行无效
同一条路由、同样的请求体、同一个 scope，对 member 绑定的服务（serviceId 形如
member-wf-…）不生效：fin_budget_variance_monitor_optimizedmember-wf-…send步骤仍waitingmember 那一侧：调用不报错、无返回体，但 observatory 里该步骤的
outcome保持waiting，toolApproval四元组依旧在，运行不向前推进。也就是说请求被接受但没有产生效果——这比返回一个明确的错误更难排查。如果 member 绑定服务本就不支持这条路由，希望能返回明确错误而不是静默无效；如果应当支持，那就是一个 bug。
关于我先前的判断
这一项我在本 issue 上先后给出过四个说法，前三个都是错的：
/services/下，我一直在试错误的路径第 3 条尤其应当避免：我拿一个中间返回码当终局证据，没把调用做完就发了结论。
对因此产生的无效排查，抱歉。本条的两个事实——正确路由已验证可用、member 绑定服务放行无效——均来自实测，不是推断。
补充：一次服务解析不一致
同一轮测试中还遇到一次：绑定成功后立即 invoke，返回
而此时查询 scope 下的服务列表，该服务存在且
invokeReady: true。稍后重试即成功。看起来是注册与调用路径之间的传播不一致（与 #2999 同类，方向相反）。该次失败发生在创建 run 之前，无任何步骤执行，故无副作用。jianwei-su commentedon Aug 24, 2026
08-24：
code_execution_route_repair_unverified—— 连续两次成功后，定时任务在绑定阶段被准入拒绝FIN-02（预算监控）的每周定时任务今天首次自然触发失败。此前该定义在 08-17 与 08-18 连续两次完整跑通并发出卡片，定义、脚本、凭据均未改动。
时间线
错误
已在同一环境下手动复现（
submit=false，零副作用），错误一致。影响
code_execute与本 issue 既有记录的关系
这是
code_execute准入侧的又一个错误码。此前记录过：NYXID_PROXY_HTTP_502code_execution_credential_unavailablecode_execution_route_unavailableCODE_EXECUTION_ROUTE_POLICY_MISMATCH (contract_drift)CODE_EXECUTION_ROUTE_REPAIR_UNVERIFIED值得注意的是措辞里的 "route repair" —— 似乎平台侧存在某种路由修复流程，而本次校验未通过。若这与 08-11/08-12 那轮路由问题的修复动作有关，这个错误码可能指向修复状态未收敛，而非我方定义变化。
我方已排除
whoami在同一环境可用）CONFIRM=yes已设置，复现时同样失败）失败发生在 preview 成功之后的绑定阶段，因此不是准入前置条件缺失。
可提供
完整日志、preview 响应、复现步骤。scopeId / workflowId 可私下提供。
jianwei-su commentedon Aug 25, 2026
08-25：影响面比昨日报告更大——两个不同工作流均被拒，dry-run 同样进不去
昨天（08-24）报告的
CODE_EXECUTION_ROUTE_REPAIR_UNVERIFIED当时只观察到 FIN-02 一个工作流。今天复测后需要更正影响范围。今日实测（
submit=false，零副作用）code_execute）CODE_EXECUTION_ROUTE_REPAIR_UNVERIFIEDcode_execute）两者定义不同、调用点数不同、业务逻辑无关，错误完全一致：
{ "code": "EXTERNAL_WORKFLOW_CAPABILITY_NOT_READY", "message": "External workflow capability admission failed: CODE_EXECUTION_ROUTE_REPAIR_UNVERIFIED: The platform code execution route repair could not be verified.", "externalCapabilityReadiness": { "status": "contract_drift", "blockers": [{ "code": "CODE_EXECUTION_ROUTE_REPAIR_UNVERIFIED" }] } }三点更正与补充
1. 不是单个工作流的问题。 影响的是该 scope 下所有使用
code_execute的工作流。2.
submit=false的 dry-run 同样被拒。 昨天的观察来自定时任务（submit=true），可能被误读为与写操作或排程相关。实际上预览模式也进不去——卡点在准入阶段，与是否产生副作用无关。3. 连续两天复现。 08-24 09:40（定时触发）、08-25 10:00（重试）、08-25 手动 dry-run ×2，五次全部同一错误。定义、脚本、凭据全程未改动，同一份定义在 08-17 与 08-18 曾连续两次完整跑通。
已排除
whoami在同一环境可用explicit-request-preview每次都成功返回全部调用点CONFIRM=yes已设置，手动 dry-run 同样失败失败发生在 preview 成功之后的绑定阶段。
status: contract_drift与历史记录的关系08-12 曾出现
CODE_EXECUTION_ROUTE_POLICY_MISMATCH，status同样是contract_drift。本次错误码不同但 status 一致，且措辞为 "route repair could not be verified" —— 若平台侧存在某个路由修复流程，这个错误可能指向该流程处于未收敛状态，而非调用方定义漂移。若确为平台侧修复未完成，希望能给出预计恢复时间；若确实需要调用方做什么（例如重新声明契约），也请说明具体动作——目前的错误文案无法据以行动。
影响
该 scope 下所有含
code_execute的工作流当前均无法绑定，包括预览。业务侧生产仍在原有系统运行，无数据风险，但迁移验证完全停滞。完整日志、preview 响应、两次 dry-run 的原始输出可提供。