diff --git a/docs/adr/0007-anthropic-messages-ingress.md b/docs/adr/0007-anthropic-messages-ingress.md index 7f9942424..2153910c3 100644 --- a/docs/adr/0007-anthropic-messages-ingress.md +++ b/docs/adr/0007-anthropic-messages-ingress.md @@ -153,9 +153,12 @@ end to end, including request-validation and upstream errors. not model-specific. Mitigation: documented clearly; adequate for budgeting/UX sizing, not for hard context-limit decisions. - **`stop_reason` fidelity for stop sequences.** `stop_sequences` are honored end to end (the - OpenAI `stop` field is mapped to Anthropic `stop_sequences`), but the canonical chat type has - no distinct "stop sequence" finish reason, so a stop-sequence-triggered completion is reported - as `stop_reason: "end_turn"` rather than `"stop_sequence"`. The output is truncated correctly; + OpenAI `stop` field is mapped to Anthropic `stop_sequences`). The canonical chat type keeps the + conservative OpenAI `finish_reason` but carries a natively-reported matched sequence as a + `stop_sequence` choice extension (same spirit as the relayed `reasoning_content`), so requests + served by the Anthropic provider report the full `stop_reason: "stop_sequence"` contract. + OpenAI-family providers conflate stop-parameter hits with natural stops in `finish_reason`, so + completions there still report `stop_reason: "end_turn"`; the output is truncated correctly, only the reason label differs. - Anthropic-specific request features routed to non-Anthropic providers degrade gracefully (dropped or approximated), consistent with Postel's law. diff --git a/docs/advanced/anthropic-messages-api.mdx b/docs/advanced/anthropic-messages-api.mdx index ca128c67c..7aee1c370 100644 --- a/docs/advanced/anthropic-messages-api.mdx +++ b/docs/advanced/anthropic-messages-api.mdx @@ -28,6 +28,32 @@ endpoint routes anywhere and is fully managed. | --- | --- | | `POST /v1/messages` | Creates a message through translated model routing. Supports streaming (`stream: true`) with Anthropic-format SSE events. | | `POST /v1/messages/count_tokens` | Returns a heuristic input token estimate. | +| `GET /v1/models` | Returns the catalog in the Anthropic list shape (`type: "model"`, `display_name`, `created_at`) when the request carries the `anthropic-version` header the Anthropic SDKs always send; OpenAI shape otherwise. | + +The Anthropic Message Batches API (`/v1/messages/batches`) is not implemented; batch +processing is available through the OpenAI-compatible `/v1/batches` endpoint. + +## Authentication + +Both credential styles work, so the official Anthropic SDKs are drop-in: + +- `Authorization: Bearer ` — GoModel's primary scheme. +- `x-api-key: ` — the Anthropic-native header, accepted as a fallback when no + `Authorization` header is present. + +```python +import anthropic + +client = anthropic.Anthropic( + api_key="", # sent as x-api-key — works as-is + base_url="https://your-gateway", +) +client.messages.create( + model="openai/gpt-4o-mini", # any configured provider's model + max_tokens=256, + messages=[{"role": "user", "content": "Hello"}], +) +``` ## Example @@ -72,12 +98,17 @@ features that have no canonical equivalent are not preserved end to end: are forwarded. - **`document` and other non-text/image content blocks** are rejected with a clear `400` error rather than silently dropped. -- **`stop_sequences`** are honored, but a stop-sequence-triggered completion reports - `stop_reason: "end_turn"` instead of `"stop_sequence"` (the output is still truncated +- **`stop_sequences`** are honored on every provider. Providers that report the + matched sequence natively (Anthropic) get the full contract back: + `stop_reason: "stop_sequence"` plus the `stop_sequence` value. OpenAI-family + providers conflate stop-parameter hits with natural stops in `finish_reason`, + so completions there report `stop_reason: "end_turn"` (output is still truncated correctly). - **`count_tokens`** returns a provider-agnostic heuristic estimate (≈ characters / 4), not a tokenizer-exact count. Use it for budgeting and UX sizing, not hard - context-limit decisions. + context-limit decisions. The same estimate seeds `usage.input_tokens` in the + streaming `message_start` event; the authoritative counts arrive in the final + `message_delta` event, which SDK accumulators prefer. For byte-exact Anthropic fidelity (including prompt-cache breakpoints), use the `/p/anthropic/v1/messages` passthrough route instead. diff --git a/docs/dev/2026-07-17_anthropic-sdk-compat-findings.md b/docs/dev/2026-07-17_anthropic-sdk-compat-findings.md new file mode 100644 index 000000000..7ad434e9f --- /dev/null +++ b/docs/dev/2026-07-17_anthropic-sdk-compat-findings.md @@ -0,0 +1,173 @@ +# Anthropic SDK drop-in compatibility — test findings + +Date: 2026-07-17 · Branch: `fix/anthropic-sdk` @ 23cdb251 · SDK: `anthropic` (Python) 0.117.0 + +Goal: verify GoModel's `/v1/messages` surface works as a **drop-in replacement** for the +Anthropic API when accessed through the official Anthropic SDK +(`anthropic.Anthropic(base_url=)`). + +Setup: gateway built from this branch, run locally with the repo `.env` credentials +(exact response cache disabled to keep probes deterministic). Test scripts live in the +session scratchpad (`sdk_compat_test.py`, `sdk_compat_phase2.py`) — 52 checks total. + +Models exercised (cheap/free tiers): `openai/gpt-4o-mini`, `gemini/gemini-2.5-flash-lite`, +`groq/llama-3.1-8b-instant` (+ `llama-3.3-70b-versatile`), `deepseek/deepseek-v4-flash`, +`anthropic/claude-haiku-4-5-20251001`, plus `/p/anthropic` passthrough. + +## Resolution status (2026-07-17, same branch) + +| Finding | Status | +| --- | --- | +| F1 x-api-key auth | **Fixed** — auth middleware falls back to `x-api-key` when no `Authorization` header (`internal/server/auth.go`). `Anthropic(api_key=...)` now works on translated and passthrough routes. | +| F2 stop_sequence lost | **Fixed for providers that report it** — the anthropic provider parses native `stop_sequence` and carries it as a `stop_sequence` choice/delta extension (same pattern as `reasoning_content`); the Messages dialect maps it back to `stop_reason: "stop_sequence"` + the matched value, stream and non-stream. OpenAI-family providers structurally can't report it (finish_reason conflates); documented. | +| F3 message_start usage=0 | **Fixed (best effort)** — `message_start` now carries the chars/4 heuristic estimate; authoritative usage still lands in `message_delta`, which SDK accumulators prefer. | +| F4 models list shape | **Fixed** — `GET /v1/models` renders the Anthropic list shape (`type`, `display_name`, `created_at`, `has_more`/`first_id`/`last_id`) when the request carries the `anthropic-version` header Anthropic SDKs always send. | +| F5 non-canonical 404 | **Fixed** — unknown routes return the canonical error envelope, Anthropic-shaped for Anthropic-dialect callers (`e.RouteNotFound` + `handleRouteNotFound`). Messages batches API itself stays unimplemented, now documented. | +| F6 cache_control dropped | **Documented** (`docs/advanced/anthropic-messages-api.mdx`) — real propagation needs cache-breakpoint representation in the canonical type; use `/p/anthropic` for prompt caching. | +| F7 heuristic count_tokens | **Documented** — passthrough gives exact counts. | +| F8 unsigned thinking blocks | **Documented** — by design; replay against the gateway works. | + +Retest after fixes: phase 1 — 32 PASS, 0 real findings (remaining two entries are the +documented OpenAI stop_sequence limitation and the documented batches gap, now with a +canonical 404 envelope). Phase 2 — all pass including x-api-key on passthrough, +`stop_reason: "stop_sequence"` + matched value on the Claude path (stream and +non-stream), and message_start input estimate. + +## Verdict + +The core Messages API contract works well across all five providers — non-streaming, +streaming (full event sequence), system prompts, multi-turn, tool use (forced choice, +roundtrip, parallel, streaming `input_json_delta`), vision, thinking, and canonical +Anthropic error envelopes. **One auth blocker and a handful of contract deviations +stand between "works" and "drop-in".** + +## Findings + +### F1 — SDK default auth (`x-api-key`) is rejected · **blocker** + +`anthropic.Anthropic(api_key=...)` sends the key in the `x-api-key` header — that is the +SDK default and what every Anthropic code sample does. GoModel's auth middleware only +reads `Authorization: Bearer` (`internal/server/auth.go`), so the request fails with +401 `missing authorization header`. Same on `/p/anthropic/...` passthrough. + +Workaround today: `anthropic.Anthropic(auth_token=...)` (sends `Authorization: Bearer`). +That is exactly the "edit your code" step a drop-in replacement is supposed to avoid. + +Suggested fix: in the auth middleware, fall back to the `x-api-key` header when no +`Authorization` header is present (keep Bearer precedence). + +### F2 — `stop_sequence` information is lost · bug + +`stop_sequences` are **applied** correctly (forwarded as OpenAI `stop`; output stops at +the sequence), but the response always reports `stop_reason: "end_turn"` and +`stop_sequence: null`. The Anthropic contract is `stop_reason: "stop_sequence"` plus the +matched sequence. Reproduced on non-stream and stream, and **also on the Anthropic +provider itself** (claude → OpenAI dialect → claude loses the info because OpenAI's +`finish_reason: "stop"` conflates natural stop with stop-sequence stop). + +Fully fixing this for OpenAI-family providers is impossible from `finish_reason` alone, +but a good heuristic exists: when the request carried `stop_sequences` and the reply text +would plausibly have continued, or — for the anthropic provider — by preserving the +native `stop_reason`/`stop_sequence` through the internal translation instead of +collapsing to `finish_reason: "stop"`. + +### F3 — streaming `message_start` carries `usage.input_tokens: 0` · deviation + +Anthropic reports real `input_tokens` in the `message_start` event; GoModel emits zeros +there and only reports usage in the final `message_delta` +(`internal/anthropicapi/stream.go: ensureStarted`). The SDK's +`get_final_message()` merges the `message_delta` usage, so SDK users see correct totals — +but clients that read usage from `message_start` (cost meters, some proxies) see 0. +Structural cause: the OpenAI upstream only delivers usage in the last chunk, so the +gateway can't know input tokens at stream start. Could be improved with the same +heuristic estimator used by `count_tokens`, or documented as a known deviation. + +### F4 — `client.models.list()` returns OpenAI-shaped objects · deviation + +`GET /v1/models` parses in the SDK (pagination works), but items lack the Anthropic +fields: `type: "model"`, `display_name`, `created_at` (RFC3339). SDK objects come back +with those attributes `None`. Display names exist in the catalog metadata already — +serving the Anthropic shape on this route when the client sends `anthropic-version` +(or on an `/v1/models` sibling for the Messages dialect) would close this. +Note: `/p/anthropic/v1/models` passthrough returns the genuine Anthropic shape and +works perfectly (10 models, `claude-sonnet-5` first). + +### F5 — Batches API absent; unknown `/v1/*` routes return non-canonical 404 · gap + +`client.messages.batches.*` hits `/v1/messages/batches` → 404 with echo's default body +`{"message": "Not Found"}`, not the Anthropic error envelope. The OpenAI-dialect +`/v1/batches` exists, but Anthropic SDK users can't reach it. Two separable items: +(a) Messages-dialect batches is unimplemented (fine to defer — document it); +(b) unknown-route 404s under `/v1/` could use the canonical error envelope so SDK +clients raise a clean typed error. + +### F6 — `cache_control` accepted but silently dropped · limitation + +`cache_control` markers on system/content blocks are tolerated (no 400 — good), but the +translation flattens system prompts to plain strings and drops the markers, so **prompt +caching never activates**, even when the request routes to the Anthropic provider. +Usage responses show no cache fields. Fine for correctness, costs money for heavy users. +Worth documenting; propagating breakpoints on the anthropic-provider path would be the +real fix. + +### F7 — `count_tokens` is heuristic · documented, keep an eye on it + +`/v1/messages/count_tokens` returns chars/4 (per ADR-0007) — e.g. 113 for a prompt the +real tokenizer counts ~90. Passthrough (`/p/anthropic/v1/messages/count_tokens`) returns +exact counts (verified: 9 tokens). Users doing budget math against the translated route +should be pointed at the passthrough. + +### F8 — thinking blocks have no `signature` · minor + +Translated responses surface reasoning as `thinking` blocks with `signature: null` +(real Anthropic thinking blocks are signed). The SDK tolerates it, and GoModel drops +incoming thinking blocks on replay (by design), so multi-turn works against the gateway. +Only a client that captures gateway output and replays it against api.anthropic.com +directly would break. Verified thinking+tool-use multi-turn roundtrip works. + +## Postel-lenient behaviors (working as intended, no action) + +- Non-alternating roles and assistant-first message arrays are accepted (Anthropic + rejects both with 400). Generous-input by design. +- `top_k` silently dropped (documented in ADR-0007 — would 400 on OpenAI-family + providers if forwarded). +- `document` blocks and server tools (`web_search_*`, …) rejected with a clear 400 + invalid_request_error pointing at the `/p/anthropic` passthrough. Good DX. +- `metadata.user_id` mapped to OpenAI `user`; `temperature`/`top_p` forwarded. + +## What passed (highlights) + +- **Response shape**: `msg_` ids, `type/role/content/stop_reason/usage` correct on all + 5 providers; `max_tokens` truncation → `stop_reason: "max_tokens"`. +- **Streaming**: full canonical event sequence (`message_start` → `content_block_start/ + delta/stop` → `message_delta` → `message_stop`) on all providers; SDK accumulation and + `get_final_message()` work; text, thinking (`thinking_delta`), and tool + (`input_json_delta`) block types all stream correctly. +- **Tools**: forced `tool_choice`, `any`, `none`, `disable_parallel_tool_use`, parallel + calls, tool_result as string / block list / `is_error`, full roundtrips on openai, + gemini, groq(70B), anthropic. +- **Thinking**: `budget_tokens` and `adaptive` accepted; deepseek's native reasoning is + correctly surfaced as Anthropic `thinking` blocks — a nice bonus the real Anthropic + SDK ecosystem understands. +- **Vision**: base64 and URL image sources (URL failures upstream relay as clean 400s). +- **Errors**: 400/401/404 all carry the canonical `{"type":"error","error":{...}}` + envelope and raise the right typed SDK exceptions (`BadRequestError`, + `AuthenticationError`, `NotFoundError`). Unknown model → 404 `not_found_error`. ✔ +- **Passthrough `/p/anthropic`**: basic, streaming, exact `count_tokens`, `models.list` + all work (auth aside, see F1). + +## Provider quirks observed (not gateway bugs) + +- `groq/llama-3.1-8b-instant` fails forced tool calls with provider-side + `tool call validation failed` (relayed correctly as 400). `llama-3.3-70b-versatile` + works through the identical path. +- OpenAI refuses to download some image URLs (e.g. Wikimedia SVG thumbs) — relayed + cleanly as `invalid_request_error`. + +## Suggested priority + +1. **F1** x-api-key auth — the single change that makes "point your SDK at GoModel" true. +2. **F2** stop_sequence preservation (at minimum on the anthropic provider path). +3. **F4** Anthropic-shaped model listing / **F3** message_start usage — nice-to-have parity. +4. **F5b** canonical 404 envelope under `/v1/`. +5. Document F6/F7 in the README (caching + token counting expectations). diff --git a/internal/anthropicapi/models.go b/internal/anthropicapi/models.go new file mode 100644 index 000000000..a1b49ad55 --- /dev/null +++ b/internal/anthropicapi/models.go @@ -0,0 +1,50 @@ +package anthropicapi + +import ( + "time" + + "github.com/enterpilot/gomodel/internal/core" +) + +// ModelsList is the Anthropic /v1/models response body. +type ModelsList struct { + Data []ModelInfo `json:"data"` + HasMore bool `json:"has_more"` + FirstID *string `json:"first_id"` + LastID *string `json:"last_id"` +} + +// ModelInfo is one model entry in the Anthropic models list. +type ModelInfo struct { + Type string `json:"type"` + ID string `json:"id"` + DisplayName string `json:"display_name"` + CreatedAt string `json:"created_at"` +} + +// FromModels renders the catalog in the Anthropic models-list shape. The full +// catalog is returned in one page: has_more is always false, so SDK +// auto-pagination terminates after a single request. +func FromModels(models []core.Model) *ModelsList { + out := &ModelsList{Data: make([]ModelInfo, 0, len(models))} + for _, model := range models { + out.Data = append(out.Data, ModelInfo{ + Type: "model", + ID: model.ID, + DisplayName: modelDisplayName(model), + CreatedAt: time.Unix(model.Created, 0).UTC().Format(time.RFC3339), + }) + } + if len(out.Data) > 0 { + out.FirstID = &out.Data[0].ID + out.LastID = &out.Data[len(out.Data)-1].ID + } + return out +} + +func modelDisplayName(model core.Model) string { + if model.Metadata != nil && model.Metadata.DisplayName != "" { + return model.Metadata.DisplayName + } + return model.ID +} diff --git a/internal/anthropicapi/request.go b/internal/anthropicapi/request.go index d968cf7ce..2f5ef2d8c 100644 --- a/internal/anthropicapi/request.go +++ b/internal/anthropicapi/request.go @@ -482,6 +482,34 @@ func EstimateInputTokens(req *MessagesRequest) int { for _, tool := range req.Tools { chars += len(tool.Name) + len(tool.Description) + len(bytes.TrimSpace(tool.InputSchema)) } + return tokensFromChars(chars) +} + +// EstimateChatInputTokens returns the same chars/4 heuristic for a canonical +// chat request. It seeds the stream converter's message_start usage, where the +// Anthropic contract expects input tokens before the upstream has reported any. +func EstimateChatInputTokens(req *core.ChatRequest) int { + if req == nil { + return 0 + } + chars := 0 + for _, msg := range req.Messages { + chars += len(core.ExtractTextContent(msg.Content)) + for _, call := range msg.ToolCalls { + chars += len(call.Function.Name) + len(call.Function.Arguments) + } + } + for _, tool := range req.Tools { + if raw, err := json.Marshal(tool); err == nil { + chars += len(raw) + } + } + return tokensFromChars(chars) +} + +// tokensFromChars converts a character count to the heuristic token estimate +// (roughly characters / 4, at least 1 for non-empty input). +func tokensFromChars(chars int) int { tokens := (chars + 3) / 4 if tokens == 0 && chars > 0 { return 1 diff --git a/internal/anthropicapi/request_test.go b/internal/anthropicapi/request_test.go index b4b4fa190..9ce7dcfe2 100644 --- a/internal/anthropicapi/request_test.go +++ b/internal/anthropicapi/request_test.go @@ -454,3 +454,56 @@ func TestToChatRequestRoundTripsAsJSON(t *testing.T) { t.Fatalf("json.Marshal(chat): %v", err) } } + +func TestEstimateChatInputTokens(t *testing.T) { + tests := []struct { + name string + req *core.ChatRequest + want int + }{ + { + name: "nil request", + req: nil, + want: 0, + }, + { + name: "messages only", + req: &core.ChatRequest{ + Messages: []core.Message{ + {Role: "system", Content: "You are terse."}, + {Role: "user", Content: "What is 2+2?"}, + }, + }, + // "You are terse." (14) + "What is 2+2?" (12) = 26 chars → ceil(26/4) = 7 + want: 7, + }, + { + name: "tool calls and tool definitions", + req: &core.ChatRequest{ + Messages: []core.Message{ + { + Role: "assistant", + ToolCalls: []core.ToolCall{ + {Function: core.FunctionCall{Name: "weather", Arguments: `{"city":"Paris"}`}}, + }, + }, + }, + Tools: []map[string]any{ + {"type": "function", "function": map[string]any{"name": "weather"}}, + }, + }, + // "weather" (7) + `{"city":"Paris"}` (16) = 23 chars, plus the + // marshaled tool definition + // `{"function":{"name":"weather"},"type":"function"}` (49 chars). + // Total 72 chars → ceil(72/4) = 18. + want: 18, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := EstimateChatInputTokens(tc.req); got != tc.want { + t.Errorf("estimate = %d, want %d", got, tc.want) + } + }) + } +} diff --git a/internal/anthropicapi/response.go b/internal/anthropicapi/response.go index fccd17c73..e2c9b645a 100644 --- a/internal/anthropicapi/response.go +++ b/internal/anthropicapi/response.go @@ -43,6 +43,15 @@ func FromChatResponse(resp *core.ChatResponse) *MessagesResponse { }) } out.StopReason = stopReasonFromFinish(choice.FinishReason, len(choice.Message.ToolCalls) > 0) + // Providers that report the matched stop sequence natively carry it as + // a choice extension; surface it per the Anthropic contract. OpenAI's + // finish_reason "stop" conflates natural and stop-parameter stops, so + // OpenAI-family providers keep reporting "end_turn". + if choice.StopSequence != "" && out.StopReason == "end_turn" { + out.StopReason = "stop_sequence" + sequence := choice.StopSequence + out.StopSequence = &sequence + } } if out.StopReason == "" { out.StopReason = "end_turn" diff --git a/internal/anthropicapi/response_test.go b/internal/anthropicapi/response_test.go index 8387df997..7ce1c5f2c 100644 --- a/internal/anthropicapi/response_test.go +++ b/internal/anthropicapi/response_test.go @@ -155,3 +155,39 @@ func TestFromChatResponseNil(t *testing.T) { t.Fatalf("FromChatResponse(nil) = %+v", resp) } } + +func TestFromChatResponseStopSequence(t *testing.T) { + resp := &core.ChatResponse{ + ID: "abc", + Model: "claude", + Choices: []core.Choice{{ + Message: core.ResponseMessage{Role: "assistant", Content: "1 2 3 "}, + FinishReason: "stop", + StopSequence: "7", + }}, + } + out := FromChatResponse(resp) + if out.StopReason != "stop_sequence" { + t.Errorf("StopReason = %q, want stop_sequence", out.StopReason) + } + if out.StopSequence == nil || *out.StopSequence != "7" { + t.Errorf("StopSequence = %v, want 7", out.StopSequence) + } +} + +func TestFromChatResponseStopSequenceDoesNotOverrideToolUse(t *testing.T) { + resp := &core.ChatResponse{ + Choices: []core.Choice{{ + Message: core.ResponseMessage{ + Role: "assistant", + ToolCalls: []core.ToolCall{{ID: "t1", Type: "function", Function: core.FunctionCall{Name: "f", Arguments: "{}"}}}, + }, + FinishReason: "tool_calls", + StopSequence: "7", + }}, + } + out := FromChatResponse(resp) + if out.StopReason != "tool_use" || out.StopSequence != nil { + t.Errorf("got stop_reason=%q stop_sequence=%v, want tool_use/nil", out.StopReason, out.StopSequence) + } +} diff --git a/internal/anthropicapi/stream.go b/internal/anthropicapi/stream.go index a369919dc..792e8229b 100644 --- a/internal/anthropicapi/stream.go +++ b/internal/anthropicapi/stream.go @@ -19,6 +19,7 @@ type chatChunk struct { Delta struct { Content string `json:"content"` ReasoningContent string `json:"reasoning_content"` + StopSequence string `json:"stop_sequence"` ToolCalls []chatToolCallDelta `json:"tool_calls"` } `json:"delta"` FinishReason string `json:"finish_reason"` @@ -58,13 +59,20 @@ func (u chatUsage) cacheRead() int { // NewStreamConverter wraps an OpenAI-style chat completion SSE stream and emits // the equivalent Anthropic Messages SSE event sequence. The returned reader // owns body and closes it on Close. -func NewStreamConverter(body io.ReadCloser, model string) io.ReadCloser { +// +// inputTokensEstimate seeds message_start's usage.input_tokens: the Anthropic +// contract reports input tokens at stream start, but the OpenAI upstream only +// delivers usage in the final chunk, so a heuristic estimate is the best +// available value there. The authoritative usage still arrives in +// message_delta, which SDK accumulators prefer. +func NewStreamConverter(body io.ReadCloser, model string, inputTokensEstimate int) io.ReadCloser { return &streamConverter{ - reader: bufio.NewReader(body), - body: body, - model: model, - buffer: streaming.NewStreamBuffer(1024), - toolBlock: make(map[int]int), + reader: bufio.NewReader(body), + body: body, + model: model, + buffer: streaming.NewStreamBuffer(1024), + toolBlock: make(map[int]int), + inputEstimate: inputTokensEstimate, } } @@ -76,16 +84,18 @@ type streamConverter struct { buffer streaming.StreamBuffer model string - started bool - blockOpen bool - blockType string - curIndex int - nextIndex int - toolBlock map[int]int - stopReason string - usage chatUsage - finalized bool - closed bool + started bool + blockOpen bool + blockType string + curIndex int + nextIndex int + toolBlock map[int]int + stopReason string + stopSequence string + inputEstimate int + usage chatUsage + finalized bool + closed bool } func (sc *streamConverter) Read(p []byte) (int, error) { @@ -178,6 +188,9 @@ func (sc *streamConverter) handleChunk(chunk *chatChunk) { for _, call := range choice.Delta.ToolCalls { sc.handleToolCall(call) } + if choice.Delta.StopSequence != "" { + sc.stopSequence = choice.Delta.StopSequence + } if choice.FinishReason != "" { sc.stopReason = stopReasonFromFinish(choice.FinishReason, len(sc.toolBlock) > 0) } @@ -268,7 +281,7 @@ func (sc *streamConverter) ensureStarted(id, model string) { "content": []any{}, "stop_reason": nil, "stop_sequence": nil, - "usage": map[string]any{"input_tokens": 0, "output_tokens": 0}, + "usage": map[string]any{"input_tokens": sc.inputEstimate, "output_tokens": 0}, }, }) } @@ -286,9 +299,14 @@ func (sc *streamConverter) finalize() { if stopReason == "" { stopReason = "end_turn" } + var stopSequence any + if sc.stopSequence != "" && stopReason == "end_turn" { + stopReason = "stop_sequence" + stopSequence = sc.stopSequence + } sc.emit("message_delta", map[string]any{ "type": "message_delta", - "delta": map[string]any{"stop_reason": stopReason, "stop_sequence": nil}, + "delta": map[string]any{"stop_reason": stopReason, "stop_sequence": stopSequence}, "usage": sc.usagePayload(), }) sc.emit("message_stop", map[string]any{"type": "message_stop"}) diff --git a/internal/anthropicapi/stream_test.go b/internal/anthropicapi/stream_test.go index bf071259f..facea5ba0 100644 --- a/internal/anthropicapi/stream_test.go +++ b/internal/anthropicapi/stream_test.go @@ -11,7 +11,7 @@ import ( // sequence of emitted Anthropic events (the decoded data: payloads). func drainConverter(t *testing.T, chatStream string) []map[string]any { t.Helper() - conv := NewStreamConverter(io.NopCloser(strings.NewReader(chatStream)), "fallback-model") + conv := NewStreamConverter(io.NopCloser(strings.NewReader(chatStream)), "fallback-model", 0) defer conv.Close() //nolint:errcheck out, err := io.ReadAll(conv) @@ -142,7 +142,7 @@ func (c *closeTracker) Close() error { // and leaked the provider connection. func TestStreamConverterCloseClosesUnderlying(t *testing.T) { body := &closeTracker{Reader: strings.NewReader("data: [DONE]\n\n")} - conv := NewStreamConverter(body, "m") + conv := NewStreamConverter(body, "m", 0) if _, err := io.ReadAll(conv); err != nil { t.Fatalf("ReadAll: %v", err) @@ -164,3 +164,72 @@ func TestStreamConverterEmptyStream(t *testing.T) { t.Fatalf("event sequence = %v, want %v", got, want) } } + +func TestStreamConverterStopSequence(t *testing.T) { + // The anthropic provider carries a natively-reported stop sequence as a + // delta extension field; the converter must surface it per the Anthropic + // contract instead of collapsing to end_turn. + chatStream := strings.Join([]string{ + `data: {"id":"chatcmpl-3","model":"claude","choices":[{"delta":{"content":"1 2 3 "},"finish_reason":null}]}`, + `data: {"choices":[{"delta":{"stop_sequence":"7"},"finish_reason":"stop"}],"usage":{"prompt_tokens":6,"completion_tokens":3}}`, + `data: [DONE]`, + "", + }, "\n\n") + + events := drainConverter(t, chatStream) + final := events[len(events)-2] + if final["type"] != "message_delta" { + t.Fatalf("expected message_delta before message_stop, got %v", final["type"]) + } + delta := final["delta"].(map[string]any) + if delta["stop_reason"] != "stop_sequence" || delta["stop_sequence"] != "7" { + t.Errorf("message_delta delta = %+v, want stop_reason=stop_sequence stop_sequence=7", delta) + } +} + +func TestStreamConverterMessageStartInputEstimate(t *testing.T) { + // message_start reports the heuristic input estimate (the upstream only + // delivers usage in its final chunk); message_delta stays authoritative. + chatStream := strings.Join([]string{ + `data: {"id":"chatcmpl-4","model":"gpt","choices":[{"delta":{"content":"hi"},"finish_reason":null}]}`, + `data: {"choices":[{"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":11,"completion_tokens":1}}`, + `data: [DONE]`, + "", + }, "\n\n") + + conv := NewStreamConverter(io.NopCloser(strings.NewReader(chatStream)), "m", 42) + defer conv.Close() //nolint:errcheck + out, err := io.ReadAll(conv) + if err != nil { + t.Fatalf("ReadAll: %v", err) + } + + var start, delta map[string]any + for block := range strings.SplitSeq(string(out), "\n\n") { + for line := range strings.SplitSeq(block, "\n") { + data, ok := strings.CutPrefix(line, "data: ") + if !ok { + continue + } + var payload map[string]any + if err := json.Unmarshal([]byte(data), &payload); err != nil { + t.Fatalf("unmarshal %q: %v", data, err) + } + switch payload["type"] { + case "message_start": + start = payload + case "message_delta": + delta = payload + } + } + } + + usage := start["message"].(map[string]any)["usage"].(map[string]any) + if usage["input_tokens"] != float64(42) { + t.Errorf("message_start usage = %+v, want input_tokens=42", usage) + } + finalUsage := delta["usage"].(map[string]any) + if finalUsage["input_tokens"] != float64(11) || finalUsage["output_tokens"] != float64(1) { + t.Errorf("message_delta usage = %+v, want real 11/1", finalUsage) + } +} diff --git a/internal/core/types.go b/internal/core/types.go index 9e21013a8..2709e01cf 100644 --- a/internal/core/types.go +++ b/internal/core/types.go @@ -118,6 +118,12 @@ type Choice struct { FinishReason string `json:"finish_reason"` Index int `json:"index"` Logprobs json.RawMessage `json:"logprobs,omitempty" swaggertype:"object"` + // StopSequence is the matched stop sequence when the provider reports one + // natively (Anthropic stop_reason "stop_sequence"). OpenAI's finish_reason + // "stop" conflates natural stops with stop-parameter hits, so this is an + // extension field: present only when the provider knows the answer, in the + // same spirit as the relayed reasoning_content extension. + StopSequence string `json:"stop_sequence,omitempty"` } // ResponseMessage represents a single assistant message in a chat response. diff --git a/internal/providers/anthropic/anthropic_test.go b/internal/providers/anthropic/anthropic_test.go index 0c2353c0c..b98e74ba8 100644 --- a/internal/providers/anthropic/anthropic_test.go +++ b/internal/providers/anthropic/anthropic_test.go @@ -283,6 +283,30 @@ func TestChatCompletion(t *testing.T) { } }, }, + { + name: "stop sequence hit carries the matched sequence", + statusCode: http.StatusOK, + responseBody: `{ + "id": "msg_stop", + "type": "message", + "role": "assistant", + "model": "claude-sonnet-4-5-20250929", + "content": [{"type": "text", "text": "1 2 3 "}], + "stop_reason": "stop_sequence", + "stop_sequence": "7", + "usage": {"input_tokens": 6, "output_tokens": 4} + }`, + expectedError: false, + checkResponse: func(t *testing.T, resp *core.ChatResponse) { + choice := resp.Choices[0] + if choice.FinishReason != "stop" { + t.Errorf("FinishReason = %q, want stop", choice.FinishReason) + } + if choice.StopSequence != "7" { + t.Errorf("StopSequence = %q, want 7", choice.StopSequence) + } + }, + }, { name: "API error - unauthorized", statusCode: http.StatusUnauthorized, @@ -430,6 +454,43 @@ data: {"type":"message_stop"} } }, }, + { + name: "stop sequence hit rides the chunk delta", + statusCode: http.StatusOK, + responseBody: `event: message_start +data: {"type":"message_start","message":{"id":"msg_stop","type":"message","role":"assistant","model":"claude-sonnet-4-5-20250929","content":[],"stop_reason":null,"usage":{"input_tokens":6,"output_tokens":0}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"1 2 3 "}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"stop_sequence","stop_sequence":"7"},"usage":{"output_tokens":4}} + +event: message_stop +data: {"type":"message_stop"} +`, + expectedError: false, + checkStream: func(t *testing.T, body io.ReadCloser) { + defer func() { _ = body.Close() }() + respBody, err := io.ReadAll(body) + if err != nil { + t.Fatalf("failed to read response body: %v", err) + } + responseStr := string(respBody) + if !strings.Contains(responseStr, `"stop_sequence":"7"`) { + t.Errorf("chunk stream should carry the matched stop sequence, got: %s", responseStr) + } + if !strings.Contains(responseStr, `"finish_reason":"stop"`) { + t.Errorf("finish_reason should stay OpenAI-conservative \"stop\", got: %s", responseStr) + } + }, + }, { name: "API error - unauthorized", statusCode: http.StatusUnauthorized, diff --git a/internal/providers/anthropic/chat.go b/internal/providers/anthropic/chat.go index eede905eb..81caa20dc 100644 --- a/internal/providers/anthropic/chat.go +++ b/internal/providers/anthropic/chat.go @@ -59,6 +59,7 @@ func convertFromAnthropicResponse(resp *anthropicResponse) *core.ChatResponse { Index: 0, Message: msg, FinishReason: finishReason, + StopSequence: resp.StopSequence, }, }, Usage: usage, diff --git a/internal/providers/anthropic/chat_stream.go b/internal/providers/anthropic/chat_stream.go index c40c09700..697a671f3 100644 --- a/internal/providers/anthropic/chat_stream.go +++ b/internal/providers/anthropic/chat_stream.go @@ -315,14 +315,20 @@ func (sc *streamConverter) convertEvent(event *anthropicStreamEvent) string { // Emit chunk if we have stop_reason or usage data if (event.Delta != nil && event.Delta.StopReason != "") || event.Usage != nil { var finishReason any + delta := map[string]any{} if event.Delta != nil && event.Delta.StopReason != "" { finishReason = sc.mapStreamStopReason(event.Delta.StopReason) + // Carry the matched stop sequence as a delta extension field: + // OpenAI's finish_reason "stop" cannot express it. + if event.Delta.StopSequence != "" { + delta["stop_sequence"] = event.Delta.StopSequence + } } var usage *anthropicUsage if sc.hasUsage { usage = &sc.usage } - return sc.formatChatChunk(map[string]any{}, finishReason, usage) + return sc.formatChatChunk(delta, finishReason, usage) } case "message_stop": diff --git a/internal/providers/anthropic/types.go b/internal/providers/anthropic/types.go index b45a59b6f..6ea9794d7 100644 --- a/internal/providers/anthropic/types.go +++ b/internal/providers/anthropic/types.go @@ -72,13 +72,14 @@ type anthropicContentSource struct { // anthropicResponse represents the Anthropic API response format type anthropicResponse struct { - ID string `json:"id"` - Type string `json:"type"` - Role string `json:"role"` - Content []anthropicContent `json:"content"` - Model string `json:"model"` - StopReason string `json:"stop_reason"` - Usage anthropicUsage `json:"usage"` + ID string `json:"id"` + Type string `json:"type"` + Role string `json:"role"` + Content []anthropicContent `json:"content"` + Model string `json:"model"` + StopReason string `json:"stop_reason"` + StopSequence string `json:"stop_sequence,omitempty"` + Usage anthropicUsage `json:"usage"` } // anthropicContent represents content in Anthropic response @@ -112,12 +113,13 @@ type anthropicStreamEvent struct { // anthropicDelta represents a delta in streaming response type anthropicDelta struct { - Type string `json:"type"` - Text string `json:"text,omitempty"` - Thinking string `json:"thinking,omitempty"` - Signature string `json:"signature,omitempty"` - PartialJSON string `json:"partial_json,omitempty"` - StopReason string `json:"stop_reason,omitempty"` + Type string `json:"type"` + Text string `json:"text,omitempty"` + Thinking string `json:"thinking,omitempty"` + Signature string `json:"signature,omitempty"` + PartialJSON string `json:"partial_json,omitempty"` + StopReason string `json:"stop_reason,omitempty"` + StopSequence string `json:"stop_sequence,omitempty"` } // anthropicModelInfo represents a model in Anthropic's models API response diff --git a/internal/server/auth.go b/internal/server/auth.go index 3d8413784..6183e413a 100644 --- a/internal/server/auth.go +++ b/internal/server/auth.go @@ -4,6 +4,7 @@ import ( "context" "crypto/subtle" "errors" + "net/http" "strings" "github.com/labstack/echo/v5" @@ -50,21 +51,11 @@ func AuthMiddlewareWithAuthenticator(masterKey string, authenticator BearerToken } } - // Get Authorization header - authHeader := c.Request().Header.Get("Authorization") - if authHeader == "" { - authErr := authenticationError(c, "missing authorization header") + token, tokenErr := requestAuthToken(c.Request()) + if tokenErr != "" { + authErr := authenticationError(c, tokenErr) return writeGatewayError(c, authErr) } - - // Extract Bearer token - const prefix = "Bearer " - if !strings.HasPrefix(authHeader, prefix) { - authErr := authenticationError(c, "invalid authorization header format, expected 'Bearer '") - return writeGatewayError(c, authErr) - } - - token := strings.TrimPrefix(authHeader, prefix) if masterKey != "" && subtle.ConstantTimeCompare([]byte(token), []byte(masterKey)) == 1 { auditlog.EnrichEntryWithAuthMethod(c, auditlog.AuthMethodMasterKey) return next(c) @@ -88,6 +79,25 @@ func AuthMiddlewareWithAuthenticator(masterKey string, authenticator BearerToken } } +// requestAuthToken extracts the caller's credential from the request. The +// primary scheme is "Authorization: Bearer "; the Anthropic-native +// "x-api-key: " header is accepted as a fallback so Anthropic SDK +// clients work without switching their auth configuration. A non-empty +// errMessage describes why no token could be extracted. +func requestAuthToken(r *http.Request) (token, errMessage string) { + if authHeader := r.Header.Get("Authorization"); authHeader != "" { + const prefix = "Bearer " + if !strings.HasPrefix(authHeader, prefix) { + return "", "invalid authorization header format, expected 'Bearer '" + } + return strings.TrimPrefix(authHeader, prefix), "" + } + if apiKey := r.Header.Get("x-api-key"); apiKey != "" { + return apiKey, "" + } + return "", "missing credentials: send 'Authorization: Bearer ' or 'x-api-key: '" +} + // applyAuthKeyResult enriches the request context and audit entry with the // authenticated managed key's identity, labels, and bound user path. func applyAuthKeyResult(c *echo.Context, authResult authkeys.AuthenticationResult, userPathHeaderName string) { diff --git a/internal/server/auth_test.go b/internal/server/auth_test.go index da8edf790..c506843eb 100644 --- a/internal/server/auth_test.go +++ b/internal/server/auth_test.go @@ -49,6 +49,7 @@ func TestAuthMiddleware(t *testing.T) { name string masterKey string authHeader string + apiKeyHeader string expectedStatus int expectedBody string }{ @@ -67,11 +68,33 @@ func TestAuthMiddleware(t *testing.T) { expectedBody: "ok", }, { - name: "missing authorization header - denies request", + name: "missing credentials - denies request", masterKey: "secret-key-123", authHeader: "", expectedStatus: http.StatusUnauthorized, - expectedBody: `{"error":{"message":"missing authorization header","type":"authentication_error","param":null,"code":null}}`, + expectedBody: `{"error":{"message":"missing credentials: send 'Authorization: Bearer ' or 'x-api-key: '","type":"authentication_error","param":null,"code":null}}`, + }, + { + name: "valid x-api-key - allows request", + masterKey: "secret-key-123", + apiKeyHeader: "secret-key-123", + expectedStatus: http.StatusOK, + expectedBody: "ok", + }, + { + name: "invalid x-api-key - denies request", + masterKey: "secret-key-123", + apiKeyHeader: "wrong-key", + expectedStatus: http.StatusUnauthorized, + expectedBody: `{"error":{"message":"invalid master key","type":"authentication_error","param":null,"code":null}}`, + }, + { + name: "authorization header takes precedence over x-api-key", + masterKey: "secret-key-123", + authHeader: "Bearer wrong-key", + apiKeyHeader: "secret-key-123", + expectedStatus: http.StatusUnauthorized, + expectedBody: `{"error":{"message":"invalid master key","type":"authentication_error","param":null,"code":null}}`, }, { name: "invalid authorization format - denies request", @@ -120,6 +143,9 @@ func TestAuthMiddleware(t *testing.T) { if tt.authHeader != "" { req.Header.Set("Authorization", tt.authHeader) } + if tt.apiKeyHeader != "" { + req.Header.Set("x-api-key", tt.apiKeyHeader) + } rec := httptest.NewRecorder() c := e.NewContext(req, rec) diff --git a/internal/server/error_support.go b/internal/server/error_support.go index bab1c3575..84a66e9cf 100644 --- a/internal/server/error_support.go +++ b/internal/server/error_support.go @@ -37,6 +37,21 @@ func writeGatewayError(c *echo.Context, gatewayErr *core.GatewayError) error { return c.JSON(gatewayErr.HTTPStatusCode(), gatewayErr.ToJSON()) } +// handleRouteNotFound renders unknown-route 404s in the caller's wire dialect +// so SDK clients raise clean typed errors instead of parsing echo's default +// {"message": "Not Found"} body. Anthropic SDK clients are recognized by the +// anthropic-version header they always send (the path itself is unclassified — +// that is what makes it a 404). +func handleRouteNotFound(c *echo.Context) error { + r := c.Request() + notFound := core.NewNotFoundError("unknown API endpoint: " + r.Method + " " + r.URL.Path) + if requestDialect(c) == "anthropic" || r.Header.Get("anthropic-version") != "" { + status, body := anthropicapi.ErrorFromGateway(notFound) + return c.JSON(status, body) + } + return c.JSON(notFound.HTTPStatusCode(), notFound.ToJSON()) +} + // requestDialect reports the ingress wire dialect classified for the request // path (e.g. "anthropic", "openai_compat"), or "" when unclassified. func requestDialect(c *echo.Context) string { diff --git a/internal/server/error_support_test.go b/internal/server/error_support_test.go index 49c5f80c8..2c35b3121 100644 --- a/internal/server/error_support_test.go +++ b/internal/server/error_support_test.go @@ -165,3 +165,59 @@ func TestHandleError_EnrichesAuditEntryWithGatewayErrorCode(t *testing.T) { t.Fatalf("entry.Data.ErrorCode = %q, want budget_exceeded", entry.Data.ErrorCode) } } + +func TestHandleRouteNotFound_AnthropicDialect(t *testing.T) { + e := echo.New() + req := httptest.NewRequest(http.MethodPost, "/v1/messages/batches", nil) + req.Header.Set("anthropic-version", "2023-06-01") + rec := httptest.NewRecorder() + c := e.NewContext(req, rec) + + if err := handleRouteNotFound(c); err != nil { + t.Fatalf("handler returned error: %v", err) + } + if rec.Code != http.StatusNotFound { + t.Fatalf("status = %d, want 404", rec.Code) + } + var body struct { + Type string `json:"type"` + Error struct { + Type string `json:"type"` + Message string `json:"message"` + } `json:"error"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if body.Type != "error" || body.Error.Type != "not_found_error" { + t.Errorf("envelope = %+v, want anthropic error envelope", body) + } + if !strings.Contains(body.Error.Message, "/v1/messages/batches") { + t.Errorf("message should name the path, got %q", body.Error.Message) + } +} + +func TestHandleRouteNotFound_OpenAIDialect(t *testing.T) { + e := echo.New() + req := httptest.NewRequest(http.MethodGet, "/v1/does-not-exist", nil) + rec := httptest.NewRecorder() + c := e.NewContext(req, rec) + + if err := handleRouteNotFound(c); err != nil { + t.Fatalf("handler returned error: %v", err) + } + if rec.Code != http.StatusNotFound { + t.Fatalf("status = %d, want 404", rec.Code) + } + var body struct { + Error struct { + Type string `json:"type"` + } `json:"error"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if body.Error.Type != "not_found_error" { + t.Errorf("envelope = %s, want OpenAI error envelope with not_found_error", rec.Body.String()) + } +} diff --git a/internal/server/handlers.go b/internal/server/handlers.go index 42746ce6d..5f646bc79 100644 --- a/internal/server/handlers.go +++ b/internal/server/handlers.go @@ -8,6 +8,7 @@ import ( "github.com/labstack/echo/v5" + "github.com/enterpilot/gomodel/internal/anthropicapi" "github.com/enterpilot/gomodel/internal/auditlog" batchstore "github.com/enterpilot/gomodel/internal/batch" "github.com/enterpilot/gomodel/internal/conversationstore" @@ -541,6 +542,17 @@ func (h *Handler) ListModels(c *echo.Context) error { } } + // The models route is shared by both wire dialects. Anthropic SDK clients + // are identified by the anthropic-version header they always send; render + // the Anthropic list shape for them, the OpenAI shape for everyone else. + if c.Request().Header.Get("anthropic-version") != "" { + var models []core.Model + if resp != nil { + models = resp.Data + } + return c.JSON(http.StatusOK, anthropicapi.FromModels(models)) + } + return c.JSON(http.StatusOK, resp) } diff --git a/internal/server/handlers_test.go b/internal/server/handlers_test.go index c8657b567..6aa304a53 100644 --- a/internal/server/handlers_test.go +++ b/internal/server/handlers_test.go @@ -2794,6 +2794,78 @@ func TestListModels(t *testing.T) { } } +func TestListModels_AnthropicDialect(t *testing.T) { + mock := &mockProvider{ + modelsResponse: &core.ModelsResponse{ + Object: "list", + Data: []core.Model{ + { + ID: "gpt-4o-mini", + Object: "model", + Created: 1721172741, + OwnedBy: "system", + Metadata: &core.ModelMetadata{DisplayName: "GPT-4o mini"}, + }, + { + ID: "gpt-4-turbo", + Object: "model", + Created: 1712361441, + OwnedBy: "system", + }, + }, + }, + } + + e := echo.New() + handler := NewHandler(mock, nil, nil, nil) + + // The anthropic-version header marks an Anthropic SDK client; the shared + // models route renders the Anthropic list shape for it. + req := httptest.NewRequest(http.MethodGet, "/v1/models", nil) + req.Header.Set("anthropic-version", "2023-06-01") + rec := httptest.NewRecorder() + c := e.NewContext(req, rec) + + if err := handler.ListModels(c); err != nil { + t.Fatalf("handler returned error: %v", err) + } + + var body struct { + Data []struct { + Type string `json:"type"` + ID string `json:"id"` + DisplayName string `json:"display_name"` + CreatedAt string `json:"created_at"` + } `json:"data"` + HasMore bool `json:"has_more"` + FirstID *string `json:"first_id"` + LastID *string `json:"last_id"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if len(body.Data) != 2 { + t.Fatalf("len(data) = %d, want 2", len(body.Data)) + } + first := body.Data[0] + if first.Type != "model" || first.ID != "gpt-4o-mini" || first.DisplayName != "GPT-4o mini" { + t.Errorf("first model = %+v", first) + } + if first.CreatedAt != "2024-07-16T23:32:21Z" { + t.Errorf("created_at = %q, want RFC3339", first.CreatedAt) + } + // Models without metadata fall back to the ID as display name. + if body.Data[1].DisplayName != "gpt-4-turbo" { + t.Errorf("fallback display_name = %q", body.Data[1].DisplayName) + } + if body.HasMore { + t.Error("has_more should be false (single page)") + } + if body.FirstID == nil || *body.FirstID != "gpt-4o-mini" || body.LastID == nil || *body.LastID != "gpt-4-turbo" { + t.Errorf("first_id/last_id = %v/%v", body.FirstID, body.LastID) + } +} + func TestListModels_MergesExposedModelsWithoutAliasProviderDecorator(t *testing.T) { catalog := &aliasesTestCatalog{ supported: map[string]bool{ diff --git a/internal/server/http.go b/internal/server/http.go index 9d78dc061..e08c15943 100644 --- a/internal/server/http.go +++ b/internal/server/http.go @@ -110,7 +110,16 @@ type ReadinessProbe interface { // New creates a new HTTP server func New(provider core.RoutableProvider, cfg *Config) *Server { - e := echo.New() + // The router-level NotFoundHandler fires only when no route matches the + // path at all, so unknown routes get a dialect-aware canonical error + // envelope while echo's 405 handling for known paths stays intact (a + // wildcard RouteNotFound route would shadow it and turn 405s into 404s). + e := echo.NewWithConfig(echo.Config{ + Router: echo.NewRouter(echo.RouterConfig{ + AllowOverwritingRoute: true, + NotFoundHandler: handleRouteNotFound, + }), + }) e.Logger = slog.Default() basePath := configuredBasePath(cfg) if basePath != "/" { diff --git a/internal/server/messages_handler.go b/internal/server/messages_handler.go index f6962b11a..b432eb7f5 100644 --- a/internal/server/messages_handler.go +++ b/internal/server/messages_handler.go @@ -120,7 +120,7 @@ func (s *translatedInferenceService) dispatchMessages(c *echo.Context, req *core result.Meta.FailoverModel, result.Stream, func(stream io.ReadCloser) io.ReadCloser { - return anthropicapi.NewStreamConverter(stream, model) + return anthropicapi.NewStreamConverter(stream, model, anthropicapi.EstimateChatInputTokens(req)) }, ) } diff --git a/tests/contract/testdata/golden/anthropic/messages_with_params.golden.json b/tests/contract/testdata/golden/anthropic/messages_with_params.golden.json index 352de9416..6723857f2 100644 --- a/tests/contract/testdata/golden/anthropic/messages_with_params.golden.json +++ b/tests/contract/testdata/golden/anthropic/messages_with_params.golden.json @@ -6,7 +6,8 @@ "message": { "content": "1, 2, ", "role": "assistant" - } + }, + "stop_sequence": "3" } ], "created": 0,