Skip to content

feat(api): abort signal for vscode-lm, gemini, mistral (SDK-specific quirks) #732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
easonLiangWorldedtech wants to merge 15 commits into from
Closed

feat(api): abort signal for vscode-lm, gemini, mistral (SDK-specific quirks) #732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
80f8538
feat(api): add RequestConfigBuilder class for SDK-agnostic request co…
easonliang28 Jun 23, 2026
9c2ed0d
docs(config-builder): enhance README with generic architecture design…
easonliang28 Jun 23, 2026
35b9b97
fix(config-builder): fix broken TOC link and simplify mergeAbortSigna…
easonliang28 Jun 23, 2026
f774650
fix: addHeaders default param and mergeAbortSignals order in RequestC…
easonliang28 Jun 26, 2026
0e6d895
feat(providers): add completePrompt signal/timeout tests for all 25 p…
easonliang28 Jun 24, 2026
a427faa
refactor(modelCache): export isAuthScopedProvider and writeModels fun…
easonliang28 Jun 24, 2026
94efadd
fix: handle pre-aborted secondary signal in mergeAbortSignals
easonliang28 Jun 26, 2026
ebe3946
refactor(api): unify completePrompt abort signal to use metadata.abor…
easonliang28 Jun 26, 2026
062df3b
test(task): strengthen abortSignal identity assertions and add sequen…
easonliang28 Jun 26, 2026
f73cc12
feat(api): add completePrompt abort signal support and tests for qwen…
easonliang28 Jun 26, 2026
1e8a6b0
fix(api-providers): fix timeout handling and abort signal issues in c…
easonliang28 Jun 26, 2026
2575f60
fix(api): improve timeout and abort signal handling across providers
easonliang28 Jun 26, 2026
b75a327
fix(api): standardize timeoutMs handling across all providers and fix…
easonliang28 Jun 27, 2026
768b42c
feat(api): wire AbortSignal to SDK providers (vscode-lm, gemini, mist…
easonliang28 Jun 26, 2026
5a6f256
fix: correct gemini abort signal test to match config.abortSignal path
easonliang28 Jun 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion src/api/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,17 @@ import {
} from "./providers"
import { NativeOllamaHandler } from "./providers/native-ollama"

/**
* Options for completePrompt — unified with ApiHandlerCreateMessageMetadata.
* Uses abortSignal (not signal) to match the metadata pattern used in stream path.
*/
export interface CompletePromptOptions extends Pick<ApiHandlerCreateMessageMetadata, "abortSignal"> {
/** Optional timeout override (ms) — falls back to provider default if omitted */
timeoutMs?: number
}

export interface SingleCompletionHandler {
completePrompt(prompt: string): Promise<string>
completePrompt(prompt: string, metadata?: CompletePromptOptions): Promise<string>
Comment on lines +47 to +53

@coderabbitai coderabbitai Bot Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Don't export timeoutMs as a shared completion contract until every provider honors it.

CompletePromptOptions.timeoutMs is now part of the public SingleCompletionHandler API, but src/api/providers/anthropic-vertex.ts (Line 299) and src/api/providers/bedrock.ts (Line 841) still ignore it and only forward abort state. That makes timeout behavior provider-dependent behind one shared interface. Either wire the override everywhere or narrow/document the contract before exporting it.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/api/index.ts` around lines 47 - 53, The shared completion contract is
exposing CompletePromptOptions.timeoutMs through SingleCompletionHandler before
all providers actually support it. Update the provider implementations
referenced by CompletePromptOptions and SingleCompletionHandler, especially
anthropic-vertex and bedrock, to honor the timeout override when completing
prompts, or otherwise narrow/remove timeoutMs from the exported API until every
provider consistently forwards it alongside abortSignal.

}

export interface ApiHandlerCreateMessageMetadata {
Expand Down
91 changes: 79 additions & 12 deletions src/api/providers/__tests__/anthropic-vertex.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -834,18 +834,22 @@ describe("VertexHandler", () => {

const result = await handler.completePrompt("Test prompt")
expect(result).toBe("Test response")
expect(handler["client"].messages.create).toHaveBeenCalledWith({
model: "claude-3-5-sonnet-v2@20241022",
max_tokens: 8192,
temperature: 0,
messages: [
{
role: "user",
content: [{ type: "text", text: "Test prompt", cache_control: { type: "ephemeral" } }],
},
],
stream: false,
})
expect(handler["client"].messages.create).toHaveBeenCalledWith(
{
model: "claude-3-5-sonnet-v2@20241022",
max_tokens: 8192,
temperature: 0,
messages: [
{
role: "user",
content: [{ type: "text", text: "Test prompt", cache_control: { type: "ephemeral" } }],
},
],
stream: false,
thinking: undefined,
},
undefined,
)
})

it("should handle API errors for Claude", async () => {
Expand Down Expand Up @@ -895,6 +899,69 @@ describe("VertexHandler", () => {
const result = await handler.completePrompt("Test prompt")
expect(result).toBe("")
})

it("should pass abort signal through to client", async () => {
handler = new AnthropicVertexHandler({
apiModelId: "claude-3-5-sonnet-v2@20241022",
vertexProjectId: "test-project",
vertexRegion: "us-central1",
})

const controller = new AbortController()
const mockCreate = vitest.fn().mockResolvedValue({
content: [{ type: "text", text: "response" }],
})
;(handler["client"].messages as any).create = mockCreate

await handler.completePrompt("test prompt", { abortSignal: controller.signal })
expect(mockCreate).toHaveBeenCalledWith(expect.objectContaining({ model: expect.any(String) }), {
signal: controller.signal,
})
})

it("should work without options (backward compatible)", async () => {
handler = new AnthropicVertexHandler({
apiModelId: "claude-3-5-sonnet-v2@20241022",
vertexProjectId: "test-project",
vertexRegion: "us-central1",
})

const mockCreate = vitest.fn().mockResolvedValue({
content: [{ type: "text", text: "response" }],
})
;(handler["client"].messages as any).create = mockCreate

const result = await handler.completePrompt("test prompt")
expect(result).toBe("response")
expect(mockCreate).toHaveBeenCalledWith(expect.objectContaining({ model: expect.any(String) }), undefined)
})

it("completePrompt should pass signal through to client", async () => {
const controller = new AbortController()
const mockCreate = vitest.fn().mockResolvedValue({
content: [{ type: "text", text: "response" }],
})
;(handler["client"].messages as any).create = mockCreate

await handler.completePrompt("test prompt", { abortSignal: controller.signal, timeoutMs: 5000 })
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({ model: expect.any(String) }),
expect.objectContaining({ signal: controller.signal, timeout: 5000 }),
)
})

it("completePrompt should pass timeoutMs when provided", async () => {
const mockCreate = vitest.fn().mockResolvedValue({
content: [{ type: "text", text: "response" }],
})
;(handler["client"].messages as any).create = mockCreate

await handler.completePrompt("test prompt", { timeoutMs: 3000 })
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({ model: expect.any(String) }),
expect.objectContaining({ timeout: 3000 }),
)
})
})

describe("getModel", () => {
Expand Down
99 changes: 91 additions & 8 deletions src/api/providers/__tests__/anthropic.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -434,14 +434,17 @@ describe("AnthropicHandler", () => {
it("should complete prompt successfully", async () => {
const result = await handler.completePrompt("Test prompt")
expect(result).toBe("Test response")
expect(mockCreate).toHaveBeenCalledWith({
model: mockOptions.apiModelId,
messages: [{ role: "user", content: "Test prompt" }],
max_tokens: 8192,
temperature: 0,
thinking: undefined,
stream: false,
})
expect(mockCreate).toHaveBeenCalledWith(
{
model: mockOptions.apiModelId,
messages: [{ role: "user", content: "Test prompt" }],
max_tokens: 8192,
temperature: 0,
thinking: undefined,
stream: false,
},
undefined,
)
})

it("should handle API errors", async () => {
Expand All @@ -464,6 +467,86 @@ describe("AnthropicHandler", () => {
const result = await handler.completePrompt("Test prompt")
expect(result).toBe("")
})

it("should pass abort signal through to client", async () => {
const controller = new AbortController()
mockCreate.mockResolvedValueOnce({ content: [{ type: "text", text: "response" }] })
await handler.completePrompt("test prompt", { abortSignal: controller.signal })
expect(mockCreate).toHaveBeenCalledWith(
{
model: mockOptions.apiModelId,
messages: [{ role: "user", content: "test prompt" }],
max_tokens: 8192,
temperature: 0,
thinking: undefined,
stream: false,
},
{ signal: controller.signal },
)
})

it("should work without options (backward compatible)", async () => {
mockCreate.mockResolvedValueOnce({ content: [{ type: "text", text: "response" }] })
const result = await handler.completePrompt("test prompt")
expect(result).toBe("response")
expect(mockCreate).toHaveBeenCalledWith(
{
model: mockOptions.apiModelId,
messages: [{ role: "user", content: "test prompt" }],
max_tokens: 8192,
temperature: 0,
thinking: undefined,
stream: false,
},
undefined,
)
})

it("should merge signal and timeout together", async () => {
const controller = new AbortController()
mockCreate.mockResolvedValueOnce({ content: [{ type: "text", text: "response" }] })
await handler.completePrompt("test prompt", { abortSignal: controller.signal, timeoutMs: 10000 })
expect(mockCreate).toHaveBeenCalledWith(
{
model: mockOptions.apiModelId,
messages: [{ role: "user", content: "test prompt" }],
max_tokens: 8192,
temperature: 0,
thinking: undefined,
stream: false,
},
expect.objectContaining({ signal: controller.signal, timeout: 10000 }),
)
})

it("should pass timeoutMs through to client alongside abortSignal", async () => {
const controller = new AbortController()
mockCreate.mockResolvedValueOnce({ content: [{ type: "text", text: "response" }] })
await handler.completePrompt("test prompt", { abortSignal: controller.signal, timeoutMs: 5000 })
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({ model: mockOptions.apiModelId }),
expect.objectContaining({ signal: controller.signal, timeout: 5000 }),
)
})

it("should pass the same signal instance", async () => {
const controller = new AbortController()
mockCreate.mockResolvedValueOnce({ content: [{ type: "text", text: "response" }] })
await handler.completePrompt("test prompt", { abortSignal: controller.signal })
expect(mockCreate).toHaveBeenCalledWith(
expect.any(Object),
expect.objectContaining({ signal: controller.signal }),
)
// Verify it's the exact same instance, not just equal
const callOptions = mockCreate.mock.calls[0][1]
expect(callOptions?.signal).toBe(controller.signal)
})

it("should not include signal-related options when not provided", async () => {
mockCreate.mockResolvedValueOnce({ content: [{ type: "text", text: "response" }] })
await handler.completePrompt("test prompt")
expect(mockCreate).toHaveBeenCalledWith(expect.any(Object), undefined)
})
})

describe("getModel", () => {
Expand Down
54 changes: 54 additions & 0 deletions src/api/providers/__tests__/base-openai-compatible-provider-timeout.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,4 +116,58 @@ describe("BaseOpenAiCompatibleProvider Timeout Configuration", () => {
}),
)
})

describe("completePrompt", () => {
it("should pass timeout through to client when both signal and timeoutMs provided", async () => {
const handler = new TestOpenAiCompatibleProvider("test-api-key")
const controller = new AbortController()
const mockCreate = vitest.fn().mockResolvedValue({
choices: [{ message: { content: "response" } }],
})
handler["client"].chat.completions.create = mockCreate

await handler.completePrompt("test prompt", { abortSignal: controller.signal, timeoutMs: 5000 })
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({ model: "test-model" }),
expect.objectContaining({ signal: expect.any(AbortSignal), timeout: 5000 }),
)
})

it("should pass only timeoutMs when no signal provided", async () => {
const handler = new TestOpenAiCompatibleProvider("test-api-key")
const mockCreate = vitest.fn().mockResolvedValue({
choices: [{ message: { content: "response" } }],
})
handler["client"].chat.completions.create = mockCreate

await handler.completePrompt("test prompt", { timeoutMs: 3000 })
expect(mockCreate).toHaveBeenCalledWith(expect.objectContaining({ model: "test-model" }), { timeout: 3000 })
})

it("should handle timeoutMs=0 as valid value (!== undefined check)", async () => {
const handler = new TestOpenAiCompatibleProvider("test-api-key")
const mockCreate = vitest.fn().mockResolvedValue({
choices: [{ message: { content: "response" } }],
})
handler["client"].chat.completions.create = mockCreate

await handler.completePrompt("test prompt", { timeoutMs: 0 })
expect(mockCreate).toHaveBeenCalledWith(expect.objectContaining({ model: "test-model" }), { timeout: 0 })
})

it("should work without options (backward compatible)", async () => {
const handler = new TestOpenAiCompatibleProvider("test-api-key")
const mockCreate = vitest.fn().mockResolvedValue({
choices: [{ message: { content: "response" } }],
})
handler["client"].chat.completions.create = mockCreate

const result = await handler.completePrompt("test prompt")
expect(result).toBe("response")
expect(mockCreate).toHaveBeenCalledWith(
expect.objectContaining({ model: "test-model" }),
{}, // empty object when no options
)
})
})
})
Loading
Loading