feat(providers): support multiple credential profiles with weighted load balancing #2677

Problem

A model connection currently has one effective provider credential. When that credential reaches a rate, usage, or billing limit—or is invalid for one model—the whole connection becomes unavailable even when the user has another API key or provider account for the same endpoint and model configuration.

The current workaround is to create duplicate connections and switch manually. That duplicates endpoint/model/header configuration, loses one stable connection identity, does not provide turn-level stickiness, and cannot safely distinguish credential failover from model/provider fallback.

This is related to the multi-account key-shape question raised in #32, but that RFC only unified credential storage. It is also downstream of the failure taxonomy in #2516 / #2521: automatic account failover must not guess from raw provider error text.

Desired outcome

Allow one connection to own multiple Credential Profiles (a primary profile plus optional secondary profiles), with explicit opt-in weighted load balancing.

The design should preserve these invariants:

• Existing single-credential connections remain an implicit primary profile and keep byte-for-byte legacy behavior while routing mode is disabled.
• Each profile has an independent credential locator, revision/CAS lifecycle, label, weight, enabled state, verification evidence, and health state.
• Balanced routing is explicit; adding a secondary credential must not silently enable it.
• Only enabled, configured profiles with current credential + execution-basis + model-support evidence are eligible.
• Selection uses deterministic smooth weighted round robin, with one sticky binding per turn and credential resolution repeated for each physical provider attempt.
• Credential-global failures and credential+model failures remain separate. One model permission failure must not disable the profile for unrelated models.
• Open circuits allow at most one half-open probe. Abort, failure, replacement, disable, and deletion must not leave a permanently claimed circuit.
• Failover only consumes normalized failure kinds and conservative routing scope. It must remain inactive until fix(runtime): distinguish usage limits from authentication failures #2516 / fix(runtime): distinguish usage limits from auth errors #2521 or an equivalent structured taxonomy lands.
• No credential failover occurs after visible output or tool-call side effects.
• Every physical model-call attempt records the actual selected profile and selection reason, without exposing secrets or mutable labels.
• Desktop exposes profile CRUD, credential replacement, test/discovery, readiness, cooldown state, and explicit routing activation.
• Configuration export/import uses stable export-local profile references, never exports verification/health state, imports profiles disabled, re-verifies them, and restores balanced mode only when activation gates still hold.
• Headless and OAuth-profile support can land in later, independently reviewable slices.

Acceptance criteria

• Legacy primary-only connections have no routing or retry behavior change.
• Weighted distribution and deterministic tie-breaking are covered for multiple ratios.
• Disabled, unconfigured, stale-evidence, unsupported, open, invalid, and half-open profiles are excluded correctly.
• Turn stickiness, per-attempt credential freshness, bounded retry count, single-flight probes, abort cleanup, and model-scoped health isolation have concurrency coverage.
• Balanced activation fails closed unless current verification proves at least one eligible profile for every enabled model and at least one model has two eligible profiles.
• Credential rotation, endpoint/header/protocol changes, profile removal, connection removal, config import, and process restart have lifecycle/crash-safety tests.
• API, protocol, logs, diagnostics, accounting, and exported config never leak credential material.

Alternatives or workarounds

• Duplicate connections + manual switching: works today but duplicates configuration and cannot provide safe turn binding or structured failover.
• Comma-separated API keys: rejected because it has no identity, CAS, verification, health, attribution, or safe import/export semantics.
• Random selection per request: rejected because it breaks turn consistency and makes failures/tool-call continuations difficult to reason about.
• Parsing provider error messages to rotate accounts: rejected because wording is unstable and ambiguous; unknown scope must remain fail closed.
• Quota-aware scheduling: useful only where an official provider API exposes trustworthy quota/reset data; it should be an optional enhancement, not a prerequisite for basic weighted routing.

Related work

• RFC: unify credential storage for desktop + headless (pure-Node CredentialStore) #32 — credential storage and the original multi-account key-shape question
• fix(runtime): distinguish usage limits from authentication failures #2516 — distinguish usage limits from authentication failures
• fix(runtime): distinguish usage limits from auth errors #2521 — proposed normalized provider failure taxonomy
• [Tracking] Runtime Host remote access and production hardening #2522 — standalone Runtime Host tracking; useful for later headless exposure but not a duplicate

Contribution request

I have prepared a detailed design and am willing to implement this in small, independently reviewable PRs, beginning with contracts/storage and keeping automatic failover gated on the structured failure taxonomy.

Maintainers: if this direction is acceptable, please assign this issue to @jischeng.