Skip to content

Docs audit — llm gateway (routing, tiers, fallbacks) — 2026-08-19 #82

Closed
#89
Closed
Docs audit — llm gateway (routing, tiers, fallbacks) — 2026-08-19#82
#89
Labels
docs-syncFiled by the docs-sync harnessmissing-caseA user situation is not coveredwrongDocs say it incorrectly

Description

@SebConejo
SebConejo
opened on Aug 19, 2026

Audited: the LLM gateway surface of mnfst/manifest at main 6b045749b, against the docs at ca319b8.

Scoped audit of the llm gateway topic (routing, custom tiers, direct model routing, the fallback chain, hard limits, gateway response headers), triggered by the nightly rotation (last audited 2026-08-09, issue #53).

  • Page in scope: llm-gateway.mdx. Findings on other pages are out of scope for this run.
  • Platform window: the full current gateway code path, plus the merged platform PRs of the last 30 days that touch routing, proxy, tiers, fallback and limits (#2639, #2600, #2573, #2676, #2689, #2685, #2712, #2611, #2535).
  • Docs commit: ca319b8. node scripts/check-coherence.mjs is GREEN (48 pages, 57 vars).
  • Scope note: no platform PR landed on main since the last full run, which is why the rotation ran. PR #2734 is marked merged on GitHub, but its base branch is fix/ollama-cloud-models-dev-capabilities, not main, so nothing in it is live and nothing here depends on it.

Four findings. The five findings of issue #53 were re-verified at HEAD: findings 1 to 4 shipped and hold, finding 5 (a Routing page screenshot) is still awaiting capture and is not re-raised.

Findings

1 · [WRONG] The page says twice that naming a model skips routing. A matching custom tier header overrides the model you name, and the tier's fallbacks then apply

Page: https://manifest.build/docs/llm-gateway (two places: the Routing step of How a request flows, and Route a specific model)

What actually happens:

  1. Your client sends a request that names a model, for example openai/gpt-4o.
  2. Manifest compares the request headers against your enabled custom tiers first, before it looks the model up.
  3. If a tier matches, Manifest ignores the model you named and sends the request to the tier's model.
  4. The tier's fallbacks apply too, so a failure moves down the tier's chain instead of coming back to you.
  5. The response carries X-Manifest-Tier: standard and X-Manifest-Reason: header-match, not direct.

The Custom tiers section already states the rule. The two sentences that promise the opposite were left behind.

Options:

  1. Rewrite both sentences so the exception is stated where the promise is made. Recommended (it removes the contradiction at both places).
  2. Keep both sentences as they are and add a note under Route a specific model that carries the exception.
  3. Lead the Custom tiers section with the precedence rule, and reduce the other two mentions to a cross-reference.

Suggested wording (option 1), line 21:

A custom tier matches on a request header, or the request goes to your default tier. Send a real model ID instead of auto to skip routing. A matching custom tier header still wins.

Suggested wording (option 1), line 75:

To skip routing for a single request, send a real model ID instead of auto. Manifest forwards it straight to that model's provider, with no tier lookup and no fallbacks. A matching custom tier header is the exception. The request then runs on the tier's model and the tier's fallbacks. Call GET /v1/models to list the model IDs your harness can reach.

Evidence (for the agent)
  • llm-gateway.mdx:21: "A custom tier matches on a request header, or the request goes to your default tier. Send a real model ID instead of auto to skip this entirely."
  • llm-gateway.mdx:75: "To skip routing for a single request, send a real model ID instead of auto. Manifest forwards it straight to that model's provider, with no tier lookup and no fallbacks."
  • Contradicted on the same page at llm-gateway.mdx:52: "The header wins even when the body names an explicit model ID."
  • Code: proxy.service.ts:904 sends a concrete model into resolveExplicitModel; :976-979 calls resolveHeaderTier first and returns on a match at :978, before the catalog lookup at :981. Comment at :957: "A matching header tier wins".
  • Only enabled tiers are compared (resolve.service.ts:260), in sort_order (header-tier.service.ts:63).
  • On a header match, explicitRouting() (proxy.service.ts:1058-1073, fallback_routes: null at :1066) is never reached, so explicit_model_override stays unset, the !explicitModelOverride gate at :502 passes, and tryFallbackChain runs at :507.
  • Headers on that path: X-Manifest-Tier: standard, X-Manifest-Reason: header-match, confidence 1 (resolve.service.ts:324-336). buildBaseMeta substitutes direct only when explicit_model_override is true (proxy.service.ts:1323-1329).
  • Not a re-raise. Issue Docs audit — reference (api, headers, environment variables, glossary) — 2026-08-16 #68 finding 8 asked for the missing sentence at :52 and it shipped; this is the pair of sentences that fix did not touch.
  • llm-gateway.mdx:87 ("The response comes back with X-Manifest-Tier: direct, so you can tell a direct call from a routed one") stays true and no option touches it: on a header match the response reports standard, which is exactly how the reader tells the two apart.
  • Impact: the SDK model field is mandatory, so a client that names a real model and also sends its tier header silently runs and bills the tier's model. The code names this exact case at proxy.service.ts:957-959: "A matching header tier wins: that rule is an override the operator configured on purpose, and the SDK's model field is mandatory, so most agents send a name they cannot change."
  • Related: issue Docs audit — getting started (introduction, first request) — 2026-08-18 #71 finding 1 (still open) cites llm-gateway.mdx:35,75 as the sentence that states the truth. Write the two fixes together, or introduction.mdx inherits the same over-absolute claim.

2 · [MISSING-CASE] The fallback section promises a response instead of an error, and never says that a streaming request is only protected until its first chunk

Page: https://manifest.build/docs/llm-gateway (Fallback, the What triggers a fallback section)

What actually happens:

  1. You send a streaming request and Manifest forwards it to your primary model.
  2. If the provider answers but sends nothing at all, Manifest gives up on it and moves to the next model in the chain.
  3. Once the first chunk reaches your client, the protection is over.
  4. If the stream then dies, Manifest writes the error inside the open stream and closes it. No other model is tried.
  5. Your agent is left with half an answer plus an error event.

The API reference already states the second half. The feature page does not.

Options:

  1. Add the boundary to What triggers a fallback, in the wording of the reference page. Recommended (one paragraph, and the feature page stops disagreeing with the reference).
  2. Give streaming its own short subsection under Fallback, covering the protection before the first chunk and the in-band error after it.
  3. Leave the page as it is and rely on the reference page.

Suggested wording (option 1), as the last paragraph of What triggers a fallback:

Fallback covers a streaming request until the first chunk reaches your client. If the stream fails after that, the error arrives inside the stream and the connection closes. Manifest tries no other model.

Evidence (for the agent)
  • llm-gateway.mdx:93: "When a model fails (provider outage, rate limit, bad request), Manifest retries with a backup model from the same tier. Your agent gets a response instead of an error."
  • llm-gateway.mdx:97: "Any HTTP status code >= 400 triggers a fallback." The table at :99-108 carries no streaming caveat. The page mentions streaming once, at :24 ("streams the response back"), and never inside Fallback.
  • Sibling sentence already in the corpus, reference/api.mdx:184: "Routing and fallback both work with streams. If the primary model fails before the first chunk, the request restarts on the fallback. If it fails mid-stream, the connection closes. There's no silent mid-stream retry."
  • Before the first chunk: a streaming 200 that produces nothing inside the warm-up window becomes a synthetic 502 (proxy.service.ts:592-596) that re-enters the chain (:612-634).
  • After the first chunk: proxy.controller.ts:489 sets headersSent; handleProxyError short-circuits on it at :740-748 and calls writeStreamError (:804-839), which writes an in-band SSE error event and ends the response. No fallback hop runs.
  • The warm-up window is already documented: reference/environment-variables.mdx:59 carries STREAM_WARMUP_MS, default 15000, "How long to wait for the first streamed chunk before falling back to the next model". Naming it in a self-hosted sentence is allowed. The fix does not need it: the boundary is true at any value.
  • Impact: an agent using stream: true with a configured chain can receive a half-written answer plus an error event, while the page promises a response instead of an error.
  • Findings 2 and 4 both edit What triggers a fallback. If both are accepted, ship them in one PR.

3 · [MISSING-CASE] A request whose text contains HEARTBEAT_OK skips your custom tiers and your default tier. Manifest answers it with "no providers are set up yet", even on a fully configured harness

Page: https://manifest.build/docs/llm-gateway (Routing, the Default tier section)

What actually happens:

  1. Your client sends an auto request whose last user message contains the text HEARTBEAT_OK.
  2. Manifest recognizes it and routes it apart from the tiers you configured. Your custom tiers and your default tier are both skipped.
  3. It goes to a routing slot that the current dashboard does not let you configure and that guided setup never fills.
  4. That slot is empty, so Manifest answers with M101: "You're connected, but no providers are set up yet."
  5. Everything else on the same harness keeps working normally, so the message reads as false.

Options:

  1. Fix it on the platform side: route a heartbeat to the default tier when the slot it targets is empty, and leave the page alone. Recommended (no wording can make an unconfigurable slot configurable).
  2. Add one paragraph to the Routing section naming the keyword, the behavior, and the symptom.
  3. Record the case and leave both sides as they are, betting that few clients send that keyword.

Suggested wording (option 2), after line 48:

Manifest treats an auto request as a heartbeat when its last user message contains HEARTBEAT_OK. A heartbeat does not go through your custom tiers or your default tier. It can come back with M101, "You're connected, but no providers are set up yet", even when the rest of your harness is configured.

Evidence (for the agent)
  • llm-gateway.mdx:39: "One model plus up to 5 fallbacks. Every request lands here unless a custom tier matches."
  • llm-gateway.mdx:48: "Send auto as the model, and any request that doesn't match a custom tier goes to your default."
  • Code: proxy.service.ts:918 computes the heartbeat flag, :926-927 calls resolveForTier(agentId, tenantId, 'simple') instead of resolveLazy. Both the custom tier check (resolve.service.ts:110-113) and the default tier (:116-118) live inside resolveLazy, so neither is reached. Trigger string at :1387-1419.
  • Slot unreachable from the dashboard today: complexity_routing_enabled defaults to false for new agents (1781000000000-ReAddComplexityRoutingFlag.ts, agent.entity.ts:30-31), so Routing.tsx:138-148 classifies a fresh harness as clean and :567 renders the default tier card plus header tiers only, with the complexity toggle gated at :755. The string simple does not appear in Welcome.tsx. The legacy auto-assigned column is written null at every insertion site (tier.service.ts:66,166,223).
  • Empty slot: resolve.service.ts:207-217 returns a null route when no row exists; buildResolvedRouteChain returns primaryRoute: null at :467 when a row exists with no override, no fallbacks and no legacy route.
  • Result: proxy.service.ts:275-287 takes buildNoProviderResult, :1425-1428 formats M101, and error-codes.ts:34-37 reads "You're connected, but no providers are set up yet." Delivered as an HTTP 200 assistant message.
  • The product's word for this path is "heartbeat": the Requests table icon carries title="Heartbeat" (message-table-cells.tsx:125-131). No .mdx page names it; a corpus grep for "heartbeat" returns zero hits.
  • Impact: on a fully configured harness, any auto request carrying that keep-alive string is answered by Manifest with a message that contradicts the rest of the product.
  • Constraint for the fix: do not introduce the retired tier vocabulary into this page. Its absence here is a recorded decision.
  • Option 1 is also recorded on the harness backlog as a platform item.

4 · [MISSING-CASE] After an upstream 429, Manifest stops calling that model for a while. The page lists 429 as a fallback trigger and never mentions the pause, or the 429 Manifest returns on its own

Page: https://manifest.build/docs/llm-gateway (Fallback, the What triggers a fallback section, and Route a specific model)

What actually happens:

  1. A provider rate-limits one of your models and returns 429.
  2. Manifest records that model as cooling down and stops calling it for a short time. When the provider sends a Retry-After, that value sets the pause.
  3. During the pause, a routed request goes to the next model in the chain without the provider being contacted at all.
  4. A request that named the model directly has no chain, so it comes back as a 429 that the provider never sent.
  5. The request log shows an attempt for the paused model with no provider call behind it.

Options:

  1. Add the pause and both of its outcomes to What triggers a fallback. Recommended (it names the one symptom a user cannot otherwise explain).
  2. Document only the direct-call half under Route a specific model: a named model can return 429 without reaching the provider.
  3. Leave as it is. The behavior is favourable in the routed case and clears itself within minutes.

Suggested wording (option 1), after the status table:

An upstream 429 also pauses that model for a short time. When the provider sends a Retry-After, that value sets the pause, up to five minutes. During the pause, Manifest sends the request to the next model in the chain. A request with no chain behind it, such as a direct model ID, comes back as a 429 that the provider never sent.

Evidence (for the agent)
  • llm-gateway.mdx:104: the table row "| 429 | Rate limited |" is the page's only mention of 429 on the fallback path.
  • llm-gateway.mdx:75: "Manifest forwards it straight to that model's provider, with no tier lookup and no fallbacks."
  • Code: proxy-fallback.service.ts:521-533 records the cooldown on a 429, keyed at :569-578 on agent, provider, auth type, key label and model. TTL is the provider's Retry-After when parseable, capped at 5 minutes (:96, parser at :552-567), 15 seconds otherwise (:95).
  • :392-395 returns buildRateLimitCooldownForward before any network call while the entry is live: a synthetic 429 carrying retry-after and the body Provider route temporarily cooling down after an upstream 429: ${provider}/${model} (:488-519), with an attempt row reserved at providerCallStarted: false.
  • The primary forward uses the same method (proxy.service.ts:419) and passes the agent id (:429) and auth type (:435), so the direct case is covered.
  • Impact: for up to a few minutes after one upstream 429, a direct call to that model is refused by Manifest with a status the provider never sent, and the request log shows an attempt with no provider call.
  • Constraint for the fix: the cooldown map is per process, so the pause applies per node on a multi-node deployment. Do not print the 15-second default as a guarantee.
  • Findings 2 and 4 both edit What triggers a fallback. If both are accepted, ship them in one PR.

Open questions (not findings)

  1. Does the per-attempt timeout bound the body of a streaming response, or only the time to the response headers? On the streaming path the abort timer is created at provider-client.ts:761-767 and cleared in the finally at :796-798, after fetch resolves, which suggests it bounds time to headers only. If that is so, the advice at llm-gateway.mdx:132-134 about lowering the timeout below your client's timeout is buffered-only reasoning, and finding 2's wording should say so. One runtime test against a stalling stream settles it.
  2. Does the API accept more than 5 fallbacks on a custom tier? The dashboard enforces 5 and the default tier validates it server side (routing.dto.ts:202,213), but the custom tier body has no size validator on models (header-tier.controller.ts:82-84, only routes capped at :88) and no database constraint exists. The sentence at llm-gateway.mdx:112 matches what the dashboard allows, so it stands either way. The answer only decides whether it needs a caveat.

Reply with the finding number and the chosen option
(e.g. "3 → option 2", or "3: other idea, let's discuss").
PRs are then made manually with the agent, never by the harness.
Rejected findings: reply "N → reject" and they will never be re-raised.

Activity

  1. SebConejo
    added
    docs-syncFiled by the docs-sync harness
    wrongDocs say it incorrectly
    missing-caseA user situation is not covered
    on Aug 19, 2026
  2. SebConejo
    added a commit that references this issue on Aug 19, 2026

    docs-sync: llm gateway rotation, 4 findings filed as issue #82

    83776bd

  3. SebConejo commented on Aug 20, 2026

    @SebConejo
    SebConejo
    on Aug 20, 2026
    MemberAuthor

    1 → option 1, 2 → option 1, 4 → option 1. PR #89.

    3 needs your call. A request whose last user message contains HEARTBEAT_OK skips your tiers and lands on a slot the dashboard cannot configure, so it answers M101 on a fully configured harness. The recommended option is a platform fix: route the heartbeat to the default tier when that slot is empty, and leave the docs alone. I can prepare that PR on mnfst/manifest. Option 2 is one doc paragraph naming the keyword and the symptom. Tell me which.

  4. SebConejo
    added a commit that references this issue on Aug 20, 2026

    docs-sync: record Seb's amendments on the #82/#84 fallback additions

    24ee403
  5. SebConejo
    added a commit that references this issue on Aug 28, 2026

    Merge pull request #89 from mnfst/fix/issue-82-gateway

    58c3146
  6. SebConejo
    closed this as completedin #89on Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    docs-syncFiled by the docs-sync harnessmissing-caseA user situation is not coveredwrongDocs say it incorrectly

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions