Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion llm-gateway.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Every harness has a default tier: one model plus up to five fallbacks. You set i

### Custom tiers

Custom tiers route by request header. You create a tier on the dashboard, give it a header key and value, and pin it to a model with its own fallbacks. When an incoming request carries that header, Manifest sends it to that tier's model instead of the default.
Custom tiers route by request header. You create a tier on the dashboard, give it a header key and value, and pin it to a model with its own fallbacks. When an incoming request carries that header, Manifest sends it to that tier's model instead of the default. The header wins even when the body names an explicit model ID.

The header key is yours to choose (lowercase letters, numbers, and hyphens). A few names are reserved and rejected, including `authorization`, `cookie`, and `x-api-key`. Send the header from your client like any other:

Expand Down
14 changes: 7 additions & 7 deletions reference/api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ curl -X POST http://localhost:2099/v1/chat/completions \
}'
```

The body is forwarded verbatim to the resolved provider, with `model` rewritten to the actual model ID. All standard OpenAI fields (`temperature`, `max_tokens`, `tools`, `tool_choice`, `response_format`, `stream`, etc.) pass through.
Manifest adapts the body to the provider it picked. It replaces `model` with the real model ID. It converts the body to the format that provider expects. It renames parameters the provider spells differently, like `max_tokens` and `max_completion_tokens`. And it merges the model parameters saved on your harness into every attempt. Some fields only exist at OpenAI: `stream_options`, `reasoning_effort`, `modalities`, `audio`, and `prediction`. Those reach OpenAI and OpenRouter. Every other provider gets the request without them, and nothing warns you.

## Anthropic messages

Expand All @@ -77,7 +77,7 @@ curl http://localhost:2099/v1/models \
-H "Authorization: Bearer mnfst_YOUR_KEY_HERE"
```

Send `auto` to let Manifest route, or send any listed model ID to skip routing and go straight to that provider. If you send a model ID that is not in this harness-specific list, Manifest returns [M302: Model not available](/errors/M302). See [Routing → Route a specific model](/llm-gateway#route-a-specific-model).
Send `auto` to let Manifest route, or send any listed model ID to skip routing and go straight to that provider. If you send a model ID that no connected provider can serve, Manifest returns [M302: Model not available](/errors/M302). See [Routing → Route a specific model](/llm-gateway#route-a-specific-model).

### Inspect model capabilities

Expand Down Expand Up @@ -185,7 +185,7 @@ Routing and fallback both work with streams. If the primary model fails before t

## Errors

The proxy returns a standard JSON error envelope:
Errors come in two shapes, depending on the caller. A tool or SDK call gets a real HTTP status and the JSON envelope below. A chat or streaming client gets an HTTP `200` that looks like a normal completion, and the error text sits inside the assistant message.

```json
{
Expand All @@ -203,17 +203,17 @@ The proxy returns a standard JSON error envelope:
|---|---|
| `401` | Invalid or missing `Authorization` header |
| `402` | Manifest Free plan quota reached (`error.code = PLAN_LIMIT_REQUESTS`, [M204](/errors/M204)), or provider billing/quota error from the upstream |
| `429` | Manifest rate limit tripped ([M201](/errors/M201), [M202](/errors/M202), [M203](/errors/M203)) |
| `429` | Manifest rate limit tripped ([M201](/errors/M201), [M202](/errors/M202), [M203](/errors/M203)), or a rate limit the provider itself returned |
| `5xx` | Upstream provider error (triggers [fallback](/llm-gateway#fallback)) |

A [hard limit](/llm-gateway#hard-limits) block ([M200](/errors/M200)) is not in this table: it comes back as an HTTP `200` chat completion whose assistant message carries the block text. When the fallback chain is exhausted, the response keeps the primary model's real error status and carries `X-Manifest-Fallback-Exhausted: true`; the body keeps the provider's own error code, or `fallback_exhausted` when there is none.

## Rate limits

Self-hosted instances default to **100 requests per 60 seconds** per harness. Override with `THROTTLE_TTL` and `THROTTLE_LIMIT` ([Environment variables](/reference/environment-variables)).
The gateway enforces three caps. A workspace can send **200 requests per minute** ([M201](/errors/M201)). An IP can send **500 requests per minute** ([M202](/errors/M202)). A workspace can have **10 requests in flight at once** ([M203](/errors/M203)). The caps are the same on Cloud and self-hosted. No setting changes them.

Cloud rate limits are tied to your plan and shown in the dashboard.
Your Cloud plan sets your monthly request quota ([M204](/errors/M204)). The per-minute caps stay the same on every plan.

## Response headers

Every response carries [routing headers](/reference/headers) so your client can see which model and tier handled the request, without parsing the response body.
Routed responses carry [routing headers](/reference/headers). They tell your client which model and tier handled the request, with no need to parse the response body. A request rejected before routing (bad key, quota, rate limit) carries none.
4 changes: 2 additions & 2 deletions reference/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ Manifest reads its configuration from environment variables. In the bundled Dock
| `MANIFEST_ENCRYPTION_KEY` | `BETTER_AUTH_SECRET` | Key used to encrypt stored provider credentials at rest. Falls back to `BETTER_AUTH_SECRET` when unset — set a separate 32+ char value so a session-cookie leak doesn't also decrypt every stored provider key. Set it before first boot |
| `MANIFEST_DISABLE_HSTS` | — | Set `1` to silence the boot warning about HSTS being unavailable on a plain-HTTP deployment. Prefer an `https://` `BETTER_AUTH_URL` anywhere reachable from the internet |
| `WINGMAN_CORS_ORIGINS` | — | Extra browser origins allowed to call the gateway, comma-separated. Useful when you host your own copy of [Wingman](https://github.com/mnfst/wingman), the open-source gateway tester; the hosted one at `wingman.manifest.build` is always allowed |
| `THROTTLE_TTL` | `60000` | Rate-limit window in milliseconds |
| `THROTTLE_LIMIT` | `100` | Max requests per window per harness |
| `THROTTLE_TTL` | `60000` | Window (ms) for the rate limit on the dashboard's own API calls. Does not apply to gateway traffic on `/v1` |
| `THROTTLE_LIMIT` | `100` | Max dashboard API calls per window. The gateway has its own fixed caps, see [Rate limits](/reference/api#rate-limits) |

## Database

Expand Down
6 changes: 3 additions & 3 deletions reference/glossary.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,19 +38,19 @@ The model ID you send to opt into routing. When Manifest sees `auto`, it applies

## Custom tier

A routing rule matched on a request header. You choose the header key and value on the dashboard and pin the tier to its own model and fallback chain. When a request carries that header, it routes to the tier's model instead of the default. See [Routing → Custom](/llm-gateway#custom-tiers).
A routing rule matched on a request header. You choose the header key and value on the dashboard and pin the tier to its own model and fallback chain. When a request carries that header, it routes to the tier's model instead of the default. That holds even when the body names an explicit model ID. See [Routing → Custom](/llm-gateway#custom-tiers).

## Direct routing

Sending a real model ID (one that `GET /v1/models` lists) instead of `auto`. Manifest forwards the request straight to that provider and skips tiers and fallbacks. The response carries `X-Manifest-Tier: direct`.
Sending a real model ID (one that `GET /v1/models` lists) instead of `auto`. Manifest forwards the request straight to that provider and skips tiers and fallbacks. The response carries `X-Manifest-Tier: direct`. One exception: if the request also carries a matching custom-tier header, the header wins.

## Fallback

The retry mechanism that kicks in when the primary model fails. Manifest tries the next model in the tier's fallback list, then the next, until one succeeds or the list is exhausted. See [Fallback](/llm-gateway#fallback) for triggers and config.

## Fallback chain

The ordered list of models tried for a single tier, primary first. Up to 5 models. When all of them fail, Manifest returns HTTP `424` with `X-Manifest-Fallback-Exhausted: true`.
The ordered list of models tried for a single tier, primary first. Up to 5 models. When all of them fail, Manifest returns the primary model's real error status, plus the header `X-Manifest-Fallback-Exhausted: true`.

## Provider

Expand Down
12 changes: 7 additions & 5 deletions reference/headers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,25 @@ Manifest reads a few request headers, and returns a set of `X-Manifest-*` respon
| ----------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization` | `Bearer mnfst_<key>` | **Required.** Authenticates the harness. |
| `Content-Type` | `application/json` | **Required.** |
| `anthropic-version` | `2023-06-01` | **Required for `/v1/messages`.** Forwarded to the upstream. |
| `anthropic-version` | `2023-06-01` | Accepted and ignored. Anthropic SDKs send it on their own. Manifest picks the version it sends upstream. |
| Your custom tier header | your value | Routes the request to a [custom tier](/llm-gateway#custom-tiers). You set the key and value when you create the tier, so the exact header name is up to you. |
| `x-session-key` | any string | Groups requests into a session. Manifest keeps a session on the same upstream where the provider supports it, for sticky routing and prompt caching. |

The custom tier header is the only one that changes routing: send the header you configured on a tier and the request goes to that tier's model. See [Routing → Custom](/llm-gateway#custom-tiers).
The custom tier header is the only one that changes routing: send the header you configured on a tier and the request goes to that tier's model. The header wins even when the body names an explicit model ID. See [Routing → Custom](/llm-gateway#custom-tiers).

## Response headers (every request)
## Response headers (routed requests)

| Header | Description | Example |
| ---------------------------- | -------------------------------------------------------------------------------- | --------------- |
| `X-Manifest-Tier` | The tier that handled the request: `default`, a custom tier's name, or `direct` | `default` |
| `X-Manifest-Tier` | How the request was routed: `standard` after a custom-tier header match, `direct` for an explicit model ID, `simple` when Manifest answered without calling a provider | `direct` |
| `X-Manifest-Model` | Model that actually served the response | `claude-sonnet-4-6` |
| `X-Manifest-Provider` | Upstream provider | `anthropic` |
| `X-Manifest-Reason` | Why that route was picked: `default`, `header-match`, or `direct` | `header-match` |
| `X-Manifest-Reason` | Why that route was picked, for example `header-match`, `direct`, or `default`. Other values exist | `header-match` |
| `X-Manifest-Output-Modality` | Output modality of the response | `text` |
| `X-Manifest-Response-Mode` | Whether the response was streamed or `buffered` | `buffered` |

Your custom tier's name appears on the request in the dashboard, never in a header. When Manifest answers a request itself (a spend block, no provider connected, a model it cannot serve), `X-Manifest-Model` and `X-Manifest-Provider` both report `manifest`.

## Response headers (fallback only)

When the primary model fails and Manifest succeeds on a fallback, two extra headers are added:
Expand Down
Loading