Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions llm-gateway.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "LLM Gateway"
description: "Every request passes through the gateway: limits check it, routing picks the model, and fallback catches it when that model fails."
icon: "split"
keywords:
["LLM gateway", "model routing", "LLM router", "fallback chain", "hard limit", "custom tier", "default model", "424 fallback exhausted", "429 too many requests"]
["LLM gateway", "model routing", "LLM router", "fallback chain", "hard limit", "custom tier", "default model", "fallback exhausted", "429 too many requests"]
---

Point your client at Manifest instead of a provider, and every request goes through the gateway. Three things happen there: limits decide whether the request runs at all, routing picks the model, and fallback steps in when that model fails.
Expand Down Expand Up @@ -84,17 +84,17 @@ curl -X POST https://app.manifest.build/v1/chat/completions \
}'
```

The response comes back with `X-Manifest-Tier: direct`, so you can tell a direct call from a routed one. Direct model IDs work on the OpenAI-format endpoints (`/v1/chat/completions` and `/v1/responses`). Requests to the Anthropic `/v1/messages` endpoint always route through your default or custom tiers.
The response comes back with `X-Manifest-Tier: direct`, so you can tell a direct call from a routed one. A direct model ID works on all three endpoints: `/v1/chat/completions`, `/v1/responses`, and the Anthropic `/v1/messages`.

The model list is scoped to the harness key. A model may be missing because its provider is not connected yet, or because the provider exists in your workspace but is not enabled for this harness. If you send an unlisted model ID, Manifest returns [M302: Model not available](/errors/M302). Send `auto` to use routing.
The model list is scoped to the harness key. A model may be missing because its provider is not connected yet, or because the provider exists in your workspace but is not enabled for this harness. If you send a model ID that has no connected provider route, Manifest returns [M302: Model not available](/errors/M302). Send `auto` to use routing.

## Fallback

When a model fails (provider outage, rate limit, bad request), Manifest retries with a backup model from the same tier. Your agent gets a response instead of an error.

### What triggers a fallback

Any HTTP status code **>= 400** triggers a fallback, with one exception: **424 (Failed Dependency)** does not, since that's the status Manifest itself returns when the entire chain is exhausted. Retrying it would loop forever.
Any HTTP status code **>= 400** triggers a fallback.

| Status | Example |
|--------|---------|
Expand Down Expand Up @@ -135,11 +135,11 @@ If your upstream client has its own timeout that fires at the same time, the cli

### Response headers

When a fallback succeeds, the response carries `X-Manifest-Fallback-From` (the primary that failed) and `X-Manifest-Fallback-Index` (its position in the chain) on top of the standard routing headers. When the chain is exhausted, `X-Manifest-Fallback-Exhausted: true` is set and the request returns `424`. Full table: [Headers reference](/reference/headers).
When a fallback succeeds, the response carries `X-Manifest-Fallback-From` (the primary that failed) and `X-Manifest-Fallback-Index` (its position in the chain) on top of the standard routing headers. When the chain is exhausted, `X-Manifest-Fallback-Exhausted: true` is set, the body carries the provider's own error code (or `fallback_exhausted` when there is none), and the response keeps the primary model's real error status (for example 500 or 529). Full table: [Headers reference](/reference/headers).

## Hard limits

A limit rule can block requests as well as email you. When a rule's action includes blocking and the harness is over its threshold for the current period, the gateway rejects the request with [M200](/errors/M200) and HTTP `429`:
A limit rule can block requests as well as email you. When a rule's action includes blocking and the harness is over its threshold for the current period, the gateway returns [M200](/errors/M200) as an HTTP `200` response whose assistant message carries the block text, so your agent keeps flowing instead of throwing:

```text
[🦚 Manifest M200] You hit your cost limit: $12.50 used, $10.00/day allowed. Adjust it here: https://app.manifest.build/...
Expand Down
15 changes: 8 additions & 7 deletions reference/api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "API"
description: "The Manifest proxy speaks both OpenAI and Anthropic. Endpoints, auth, streaming, and error responses."
icon: "code"
keywords:
["chat completions", "responses API", "Anthropic messages", "auto", "v1/models", "streaming", "SSE", "424 fallback exhausted"]
["chat completions", "responses API", "Anthropic messages", "auto", "v1/models", "streaming", "SSE", "fallback exhausted"]
---

Manifest exposes both OpenAI and Anthropic-format endpoints on one proxy. Point your client at the Manifest URL, send `auto` as the model, and routing picks the real model behind the scenes.
Expand Down Expand Up @@ -190,22 +190,23 @@ The proxy returns a standard JSON error envelope:
```json
{
"error": {
"message": "Limit exceeded: cost usage ($1.23) exceeds $1.00 per day",
"type": "limit_exceeded",
"code": 429
"message": "[🦚 Manifest M005] I don't recognize this key. ...",
"type": "auth_error",
"code": "manifest_auth"
}
}
```

`error.code` is a string identifier, never the numeric HTTP status.

| Status | Meaning |
|---|---|
| `401` | Invalid or missing `Authorization` header |
| `402` | Manifest Free plan quota reached (`error.code = PLAN_LIMIT_REQUESTS`, [M204](/errors/M204)), or provider billing/quota error from the upstream |
| `424` | Fallback chain exhausted (all configured models failed) |
| `429` | [Hard limit](/llm-gateway#hard-limits) hit ([M200](/errors/M200)), or Manifest rate limit tripped |
| `429` | Manifest rate limit tripped ([M201](/errors/M201), [M202](/errors/M202), [M203](/errors/M203)) |
| `5xx` | Upstream provider error (triggers [fallback](/llm-gateway#fallback)) |

Status `424` is the only one that **does not** trigger a fallback. Manifest returns it itself when the chain is exhausted, so re-routing it would loop forever.
A [hard limit](/llm-gateway#hard-limits) block ([M200](/errors/M200)) is not in this table: it comes back as an HTTP `200` chat completion whose assistant message carries the block text. When the fallback chain is exhausted, the response keeps the primary model's real error status and carries `X-Manifest-Fallback-Exhausted: true`; the body keeps the provider's own error code, or `fallback_exhausted` when there is none.

## Rate limits

Expand Down
2 changes: 1 addition & 1 deletion reference/headers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ When **every** model in the chain fails:

| Header | Description |
| ------------------------------- | ---------------------------------------- |
| `X-Manifest-Fallback-Exhausted` | Set to `true`. Response status is `424`. |
| `X-Manifest-Fallback-Exhausted` | Set to `true`. The response keeps the primary model's real error status; the body carries code `fallback_exhausted`. |

## Reading headers in code

Expand Down
11 changes: 6 additions & 5 deletions references/decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ The issue-verifier reads this to never re-raise a rejected finding.
- 2026-08-07 · MISSING-CASE · introduction.mdx · free plan quota (10k/month) stated nowhere outside the M204 error page · status: accepted (option 1) → PR #50 · https://github.com/mnfst/docs/issues/49 (finding 6)
- 2026-08-08 · SCOPED RUN (autofix, cron, autonomous, no human) · pages autofix.mdx + errors.mdx · window: full feature surface (PRs #2379→#2694) · CLEAN — 0 findings, no issue filed. Auditor + fresh-eyes verifier both re-verified every falsifiable claim against code: repairable statuses 400,404,422; synthetic-M302 heal (#2569) gone → forward-to-provider 404 (#2639) matches autofix.mdx:30; before-fallback/one-retry; streaming before-first-byte; toggle defaults ON-cloud/OFF-selfhosted with the nullable-reset migration (pre-existing agents inherit, not force-flipped) matching the user-visible "keeps its state"; consent self-hosted-only on any enable path; guardrails 3-failures/30s/one-success; env table (5 vars, defaults, 256 KB drop, AUTOFIX_URL is a constant not a var); dashboard cards. No internal-name leak (Phoenix/Peacock/Doctor/autofix.manifest.build absent). All 4 images present. lastAuditedAt stamped 2026-08-08; lastAuditedPr untouched (scoped run).
- 2026-08-09 · SCOPED RUN (llm gateway: routing, tiers, fallbacks · cron rotation, autonomous, no human) · page llm-gateway.mdx · window: last 30 days + full current routing/gateway code path · 5 findings filed as issue #53; lastAuditedPr untouched (scoped run), topic stamped 2026-08-09. Categories: WRONG ×3, MISSING-CASE ×1, IMAGE ×1.
- 2026-08-09 · WRONG · llm-gateway.mdx:87 · "/v1/messages always routes through tiers" false — a concrete model ID is a direct route on all three endpoints (proxy.service.ts:899-916) · status: raised · https://github.com/mnfst/docs/issues/53 (finding 1)
- 2026-08-09 · WRONG · llm-gateway.mdx:97,138,6 · invented "424 exception" to fallback + "exhaustion returns 424" false — trigger is literally >=400 no exception, exhaustion preserves the PRIMARY model's real status and signals via X-Manifest-Fallback-Exhausted + body code fallback_exhausted (fallback-status-codes.ts:1-3, proxy.service.ts:1258-1265, proxy-response-handler.ts:403-410) · status: raised · https://github.com/mnfst/docs/issues/53 (finding 2)
- 2026-08-09 · WRONG · llm-gateway.mdx:142 · "M200 hard-limit block returns HTTP 429" false — delivered as an HTTP 200 friendly assistant stub (proxy-friendly-response.ts:115-119); 429 is M201/202/203, 402 is M204 · status: raised · https://github.com/mnfst/docs/issues/53 (finding 3)
- 2026-08-09 · MISSING-CASE (low severity, honest leave-as-is option) · llm-gateway.mdx:89 · "any unlisted model ID → M302" is a false universal — an uncatalogued-but-connected model is forwarded to the provider (#2639, proxy.service.ts:981-1049); harm is missed capability, not failure. NOT a re-raise of decisions.md self-hosted.mdx M302 line (different claim, different page) · status: raised · https://github.com/mnfst/docs/issues/53 (finding 4)
- 2026-08-09 · IMAGE · llm-gateway.mdx:114-124 · Fallback Configuration steps describe the Routing page drag-to-reorder flow in prose only; screenshot of a tier + ordered fallback list missing · status: raised · https://github.com/mnfst/docs/issues/53 (finding 5)
- 2026-08-09 · WRONG · llm-gateway.mdx:87 · "/v1/messages always routes through tiers" false — a concrete model ID is a direct route on all three endpoints (proxy.service.ts:899-916) · status: accepted (fix applied, batch 2026-08-15) → PR pending · https://github.com/mnfst/docs/issues/53 (finding 1)
- 2026-08-09 · WRONG · llm-gateway.mdx:97,138,6 · invented "424 exception" to fallback + "exhaustion returns 424" false — trigger is literally >=400 no exception, exhaustion preserves the PRIMARY model's real status and signals via X-Manifest-Fallback-Exhausted + body code fallback_exhausted (fallback-status-codes.ts:1-3, proxy.service.ts:1258-1265, proxy-response-handler.ts:403-410) · status: accepted (option 1, batch 2026-08-15) → PR pending · https://github.com/mnfst/docs/issues/53 (finding 2)
- 2026-08-09 · WRONG · llm-gateway.mdx:142 · "M200 hard-limit block returns HTTP 429" false — delivered as an HTTP 200 friendly assistant stub (proxy-friendly-response.ts:115-119); 429 is M201/202/203, 402 is M204 · status: accepted (fix applied, batch 2026-08-15) → PR pending · https://github.com/mnfst/docs/issues/53 (finding 3)
- 2026-08-09 · MISSING-CASE (low severity, honest leave-as-is option) · llm-gateway.mdx:89 · "any unlisted model ID → M302" is a false universal — an uncatalogued-but-connected model is forwarded to the provider (#2639, proxy.service.ts:981-1049); harm is missed capability, not failure. NOT a re-raise of decisions.md self-hosted.mdx M302 line (different claim, different page) · status: accepted (option 2 soften, batch 2026-08-15) → PR pending · https://github.com/mnfst/docs/issues/53 (finding 4)
- 2026-08-09 · IMAGE · llm-gateway.mdx:114-124 · Fallback Configuration steps describe the Routing page drag-to-reorder flow in prose only; screenshot of a tier + ordered fallback list missing · status: accepted (screenshot pending — needs a dashboard capture, deferred) · https://github.com/mnfst/docs/issues/53 (finding 5)
- 2026-08-10 · SCOPED RUN (observability: requests page, analytics · cron rotation, autonomous, no human) · page observability.mdx · window: observability feature surface, ~45 days (PRs incl. #2374/#2456/#2485/#2502/#2508/#2535/#2609/#2642/#2666) · 4 findings filed as issue #54; lastAuditedPr untouched (scoped run), topic stamped 2026-08-10. Categories: WRONG ×2, MISSING-CASE ×1, IMAGE ×1. Full loop ran (platform-engineer truth → docs-auditor cross-check → issue-verifier fresh-eyes). Auditor cleared two near-findings: local/custom cost shows as $0 in aggregates (spirit-correct, mechanism is NULL not literal 0 — user-visible truth holds, leave-as-is), and the four-card dashboard map (fair conceptual grouping, "error rates" served by the shipped Success rate). No cross-page backlog items surfaced (env-vars reference already carries the EMAIL_* rows, coherence holds).
- 2026-08-10 · WRONG · observability.mdx:98 · self-hosted "Configure a provider on the Limits page: Resend/Mailgun/SendGrid … then save" is false — the in-app email-provider form is unmounted dead code (EmailProviderSetup has zero JSX usages; Limits.tsx:213 renders only read-only CloudEmailInfo); the EMAIL_* env vars (send-email.ts:30-59) are the ONLY self-hosted path · status: raised · https://github.com/mnfst/docs/issues/54 (finding 1)
- 2026-08-10 · WRONG · observability.mdx:67,79 · limit-rule "Action | Email you, block further requests, or both" implies selectable block-only; UI always emails and block is an optional toggle (LimitRuleModal.tsx:25 actionValue notify|both, no block-only emitted) · status: raised · https://github.com/mnfst/docs/issues/54 (finding 2)
Expand All @@ -59,6 +59,7 @@ The issue-verifier reads this to never re-raise a rejected finding.
- 2026-08-14 · WRONG (primary; secondary internal-implementation-leak, same fix) · errors/M201.mdx:27, M202.mdx:25, M203.mdx:25 (+ frontmatter descriptions/keywords) · step 3 tells self-hosted users to "raise RATE_MAX_REQUESTS/IP_RATE_MAX_REQUESTS/CONCURRENCY_MAX in proxy-rate-limiter.ts" — these are hardcoded compile-time module constants (proxy-rate-limiter.ts:5,6,8=200/500/10), the class reads no ConfigService/process.env, the names appear nowhere else in packages/backend/src, docker/.env.example has no rate keys, and the separate ThrottlerModule is @SkipThrottle()'d off the proxy; there is NO env/config/runtime override, so the documented fix is non-actionable and leaks an internal source path/symbol. Consumes backlog line 7 · status: accepted (option 1, batch 2026-08-15) → PR pending · https://github.com/mnfst/docs/issues/58 (finding 2)
- 2026-08-14 · MISSING-CASE (low severity, consistency, honest leave-as-is option 3) · errors.mdx:21 vs :55,:71 · the chat-vs-tool status split is stated only for the auth block ("HTTP 401 to non-chat clients") but M300 (":55 HTTP 400"), M500 (":71 HTTP 500") are given as flat statuses and M303 is lumped with M302 as "friendly proxy message"; truth: chat/streaming clients always get a friendly HTTP 200 stub, only tool/SDK clients get the real status — M300 chat200/tool400, M303 chat200/tool400 (provider-client.ts:261 BAD_REQUEST), M500 chat200/tool500, M302 is 200 for all (proxy.service.ts:1438); recommend a single umbrella sentence + splitting M302 from M303 · status: accepted (option 1 umbrella sentence, batch 2026-08-15) → PR pending · https://github.com/mnfst/docs/issues/58 (finding 3)
- 2026-08-10 · FULL RUN (cron, autonomous, no human) · window: merged PRs since #2701 = only #2703 · CLEAN — 0 in-scope findings, no issue filed. #2703 "test: protect gateway API contracts" is CI/test-only (contracts/gateway/* + docker-smoke.yml): adds HTTP contract tests codifying pre-existing gateway promises (POST /v1/chat/completions and /v1/responses available; unauthenticated non-streaming call → 401 with error.type auth_error), changes ZERO product behavior. The surface it protects is documented CORRECTLY (llm-gateway.mdx:87 direct IDs on chat/completions+responses; errors.mdx:21 "401 to non-chat clients"; M001–M005 → 401 auth_error). platform-engineer state-of-truth confirmed the auth 401 envelope {message,type:'auth_error',code:'manifest_auth'} reaches ONLY non-chat clients — a stream:true/SSE caller gets a friendly HTTP 200 instead (proxy-exception.filter.ts:38-44,140-147); the contract test posts plain JSON so it lands on the 401 path. Two real discrepancies on reference/api.mdx (424-on-exhaustion; envelope example type:"limit_exceeded",code:429) are OUT OF SCOPE (reference topic, lastAuditedAt null) and are page-echoes of falsehoods already raised for llm-gateway.mdx in #53 (findings 2 and 3) → parked in backlog.md, not re-raised. Coherence GREEN. lastAuditedPr advanced 2701→2703.
- 2026-08-15 · backlog lines for reference/api.mdx (424 echo, limit_exceeded/429 envelope) + reference/headers.mdx:47 CONSUMED by the #53 batch fix — same falsehoods, same decision, fixed in the same PR to honor never-halfway.
- 2026-08-15 · WRONG · errors.mdx + errors/M200.mdx · M200 documented as HTTP 429, platform returns HTTP 200 friendly assistant message (live-reproduced) + metric list wrongly included message-count (metrics are cost|tokens only) · status: accepted (Seb chose docs-fix over platform change; option B declined) → fixed with #58 findings · https://github.com/mnfst/docs/issues/52
- 2026-08-15 · WRONG · providers/api-key-providers.mdx + subscription-based-providers.mdx + local-models.mdx + custom-providers.mdx + errors/M100,M101,M102 · connect steps send the user to the Routing page; connection moved to the Providers sidebar section (Usage-based / Subscriptions / Local) in platform PR #2207 · status: raised · https://github.com/mnfst/docs/issues/67 (finding 1)
- 2026-08-15 · MISSING-CASE · providers/subscription-based-providers.mdx + api-key-providers.mdx · model lists are cached per connection and only change on connect or an explicit "Refresh models"; the Kimi Coding Plan renamed kimi-k3 to k3 and added k3-256k + kimi-for-coding-highspeed, so existing connections serve a dead id · status: raised · https://github.com/mnfst/docs/issues/67 (finding 2)
Expand Down