Skip to content

feat(openrouter): add catalogue-wide model support - #586

Merged
aviggiano merged 13 commits into
mainfrom
feat/584-openrouter
Aug 15, 2026
Merged

aviggiano merged 13 commits into
mainfrom
feat/584-openrouter

Conversation

@aviggiano

@aviggiano aviggiano commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • add a first-class Codex-backed OpenRouterAgent using OPENROUTER_API_KEY and https://openrouter.ai/api/v1
  • preserve arbitrary OpenRouter catalogue model identifiers end to end without a static allowlist
  • support local, Modal, benchmark, eval-history, schema, init, doctor, and workflow surfaces
  • document configuration and credential isolation, with deterministic tests and no live paid calls

Validation

  • pnpm -w format:check
  • pnpm --filter @ultrafuzz/config test (62 tests)
  • pnpm --filter @ultrafuzz/runtime typecheck
  • full pre-rebase release validation passed; current-main runtime and GitHub release gates are running

Closes #584.

Greptile Summary

The PR adds first-class OpenRouter support across configuration, generated runtime adapters, local and Modal execution, benchmark provenance, schemas, workflows, and documentation.

  • Preserves arbitrary validated OpenRouter catalogue identifiers without a static allowlist.
  • Adds API-key isolation and routes Codex-backed requests through the official OpenRouter endpoint.
  • Extends benchmark, publication, smoke-test, initialization, validation, and diagnostic surfaces.

Confidence Score: 4/5

The PR is not yet safe to merge because Doctor can still approve credential posture for a configured topology while a run-time OpenRouter topology override immediately fails preflight.

Doctor has no input for the run’s --topology-path and derives credential requirements from the configured topology, whereas start-run validates credentials against the override-expanded graph; the existing topology-override diagnostic mismatch therefore remains.

Files Needing Attention: packages/runtime/src/doctor.ts, packages/runtime/src/types.ts

Important Files Changed

Filename Overview
packages/runtime/src/doctor.ts Adds OpenRouter credential diagnostics, but the previously reported mismatch with run-time topology overrides remains.
packages/runtime/src/start-run.ts Adds expanded-graph-based OpenRouter credential preflight before run creation.
packages/runtime/src/templates/smithers/agents/openrouter.tsx Adds the isolated Codex-backed OpenRouter adapter, fixed provider endpoint, and managed credential configuration.
packages/config/src/model-profiles.ts Validates opaque OpenRouter catalogue identifiers while preserving supported punctuation and aliases.
packages/modal/src/benchmark-config-zod.ts Extends Modal benchmark contracts with matching OpenRouter provider, agent, authentication, and model constraints.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Config[OpenRouter model profile] --> Validate[Config and topology validation]
  Validate --> Graph[Expanded run graph]
  Graph --> Preflight[Credential and command preflight]
  Preflight --> Agent[Generated OpenRouterAgent]
  Agent --> Codex[Codex CLI]
  Codex --> Router[OpenRouter API]
  Graph --> Modal[Modal benchmark execution]
  Modal --> Evidence[Benchmark and publication evidence]
Loading

Reviews (12): Last reviewed commit: "fix(benchmarks): serialize the OpenRoute..." | Re-trigger Greptile

Context used:

Comment thread packages/runtime/src/doctor.ts Outdated
@aviggiano
aviggiano force-pushed the feat/584-openrouter branch 2 times, most recently from 2810161 to 308a57f Compare August 14, 2026 01:02
Comment thread packages/runtime/src/doctor.ts
Add a first-class OpenRouter agent across configuration, runtime, Modal, evaluation workflows, schemas, tests, and documentation. Preserve arbitrary OpenRouter catalogue model identifiers and isolate OPENROUTER_API_KEY credentials through the Codex-backed adapter.

Closes #584.
@aviggiano
aviggiano force-pushed the feat/584-openrouter branch from 308a57f to 4d531e6 Compare August 14, 2026 01:47
@aviggiano

Copy link
Copy Markdown
Collaborator Author

v0.1.0 integration policy — 2026-08-14

OpenRouter is an explicitly approved exception to the v0.1.0 feature freeze. This PR remains targeted to and must merge directly into main; do not retarget it to release/v0.1.0 or route it through PR #366. The final v0.1.0 candidate will be constructed from stabilized main and will inherit this merge.

This is a scope clarification, not merge authorization. Current blockers include the conflict with main, failing required release validation, and the unresolved Doctor versus runtime --topology-path credential-preflight concern. Resolve or explicitly disposition those blockers and obtain human review before merge.

Merging this PR would not itself authorize a live paid OpenRouter run, the v0.1.0 benchmark, tagging, or publishing.

@aviggiano

Copy link
Copy Markdown
Collaborator Author

Note: we're waiting on OpenRouter admin to remove sensitive filters, otherwise we can't test this

aviggiano and others added 11 commits August 14, 2026 18:30
Every one-shot validator worker eagerly compiled all 65 registered schemas
before validating a single root. Ajv compilation of the full bundle costs
~540ms against ~39ms for one root, so each `validateJsonFile` call paid a
~14x setup penalty. On the four-core CI runner that pushed four Modal
package-gate tests past their existing timeouts.

The eager pass was added to keep referenced-schema diagnostics stable, but
main fixed the same diagnostics drift in #595 by correcting the expected
schema paths, so the compile-all is now redundant. Ajv already compiles a
root together with its reachable references on first `getSchema`, and the
host and one-shot workers project identical paths without it.

Modal package gate on four pinned cores: 561 passed in 97s, versus four
timeouts in 364s before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OpenRouter rate-limits the routed Codex model hard enough that the smoke
lane's six concurrent CodexAgent calls exhaust Codex's HTTP retries with
429 responses before any target starts. Bound an OpenRouter runner to one
eval row and one workflow node; every other provider keeps the lane's
checked-in concurrency.

The bound is a single checked-in constant so that all four trusted policy
derivations agree: the Modal worker's suite, the producer manifest, the
launch guardrail, and the cleanup and publication re-derivations. Cleanup
now takes a policy-dimension resolver so it reads the dispatched provider
off the manifest exactly as the launch guardrail does, instead of failing
validation and leaving Modal sandboxes running.

Also drop the temporary OpenRouter diagnostic probe job and scripts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@aviggiano
aviggiano merged commit 053e91e into main Aug 15, 2026
10 of 11 checks passed
@aviggiano
aviggiano deleted the feat/584-openrouter branch August 15, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate OpenRouter for v0.1.0 via main

1 participant