Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs-site/src/content/docs/guides/remote-link.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ A failed link does not silently switch to a local provider.
The Child returns a retryable `503` without forwarding the link key or request when its tunnel is failed, stopped, or not supervised.
It also keeps the key local when another process owns the tunnel port; free that port so the Child's SSH tunnel can bind it.

To confirm which Home is serving a Child, compare the `x-opencodex-served-by` header on the Child's `GET /v1/models` with the `instanceId` in the Home's `GET /healthz`. The Child passes the Home's header through unchanged, so the two match while the link works. The value is an opaque id, never a hostname; see [Home identity and the served-by header](/reference/management-api/#home-identity-and-the-served-by-header).

## What a Child shows and manages

A Child runs on the Home's providers, accounts and settings, so the Home is where you change them.
Expand Down
17 changes: 17 additions & 0 deletions docs-site/src/content/docs/reference/management-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -658,6 +658,23 @@ API page. The CLI drives these routes with `ocx api protocols`, `ocx api explain
[API surfaces](/reference/configuration/server/#api-surfaces-apisurfaces) for how the Messages
setting interacts with `claudeCode.enabled` across upgrades and downgrades.

### Home identity and the served-by header

`GET /healthz` includes `instanceId`, this install's opaque Home id: `home_` followed by 32 hex
characters, generated at random once per install. It is the same value as `producer.homeId` in
`GET /api/quota-pool`, it stays the same across restarts, and it is never a hostname or derived from
one. Every response to `GET /v1/models` carries the same value in the `x-opencodex-served-by`
header, so a client can tell which Home answered without reading logs. A Remote Link Child relays the
Home's header unchanged, and a Hub-mode client's Codex reads the Hub's response directly, so on any
device the header names the Home that served the model list. `GET /readyz` lists
`served-by-header` in `features` when the build sends it.

Right after a fresh install starts, before its identity file exists, or when that file is
unreadable, both the `instanceId` field and the header are left out; an empty value is never sent.
Deleting `quota-pool-identity.json` from the OpenCodex home rotates the id together with every
quota-pool account reference when the proxy next starts; a running proxy keeps reporting the id it
already read.

### Providers

| Method and path | Purpose | Notable errors |
Expand Down
2 changes: 1 addition & 1 deletion scripts/test-layout/layout.json
Original file line number Diff line number Diff line change
Expand Up @@ -1453,7 +1453,7 @@
"proxy-env-macos.test.ts": "server",
"proxy-env.test.ts": "server",
"proxy-liveness-package-tree-fence.test.ts": "server",
"proxy-liveness.test.ts": "server", "readyz-features.test.ts": "server",
"proxy-liveness.test.ts": "server", "readyz-features.test.ts": "server", "served-by-header.test.ts": "server",
"qoder-adapter.test.ts": "providers", "qoder-live-models.test.ts": "providers", "qoder-mcp-server.test.ts": "providers",
"qoder-scaffold-guard.test.ts": "providers",
"quota-401-recovery-runtime.test.ts": "usage",
Expand Down
59 changes: 51 additions & 8 deletions src/remote/served-by.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,57 @@
/**
* The served-by response header and whether this build sends it.
* The served-by response header, the Home instance id it carries, and whether this build sends it.
*
* Relayed and Hub-served responses are meant to name the Home that served them in
* `x-opencodex-served-by`, so a peer can prove which Home answered without reading logs. This module
* is the emitter's descriptor: `SERVED_BY_HEADER_EMITTED` is true only in a build whose responses
* carry the header, and the change that starts sending it flips the flag in the same commit.
* `/readyz` advertises `served-by-header` from this flag (`src/remote/features.ts`), and
* `tests/server/readyz-features.test.ts` fails if the advertisement and live responses disagree.
* Every `GET /v1/models` response the server answers itself names the serving Home in
* `x-opencodex-served-by`, and `GET /healthz` publishes the same value as `instanceId`, so a peer
* (GOAT's parity check P4) can prove which Home answered without reading logs. A Remote Link Child
* never answers `/v1/models` itself: its relay copies the Home's response headers, so the value a
* Child returns is the Home's. A Hub-mode client's Codex calls the Hub's data listener directly, so
* the Hub stamps its own response.
*
* The instance id is the install's `homeId` from `src/quota-pool/identity.ts`: a random
* `home_<32 hex>` minted once per install, persisted owner-only, and stable across restarts. It is
* never a hostname and is not derived from one. Until the identity exists (it is created in the
* background after the listener starts) or when its file is unreadable, the header and the
* `/healthz` field are omitted; an empty or malformed value is never sent.
*
* `SERVED_BY_HEADER_EMITTED` tells `/readyz` to advertise `served-by-header`
* (`src/remote/features.ts`); `tests/server/readyz-features.test.ts` fails if the advertisement
* and live responses disagree.
*/
import { readQuotaPoolIdentity } from "../quota-pool/identity";

export const SERVED_BY_HEADER = "x-opencodex-served-by";

/** Whether responses from this build carry {@link SERVED_BY_HEADER}. */
export const SERVED_BY_HEADER_EMITTED: boolean = false;
export const SERVED_BY_HEADER_EMITTED: boolean = true;

const INSTANCE_ID = /^home_[0-9a-f]{32}$/;

/** This Home's opaque instance id, or undefined while the install identity is unavailable. */
export function homeInstanceId(): string | undefined {
let homeId: unknown;
try { homeId = readQuotaPoolIdentity()?.homeId; } catch { return undefined; }
return typeof homeId === "string" && INSTANCE_ID.test(homeId) ? homeId : undefined;
}

/** Stamp {@link SERVED_BY_HEADER} with this Home's instance id; leaves the response alone without one. */
export function withServedBy(response: Response, instanceId = homeInstanceId()): Response {
if (!instanceId) return response;
try {
response.headers.set(SERVED_BY_HEADER, instanceId);
return response;
} catch {
const headers = new Headers(response.headers);
headers.set(SERVED_BY_HEADER, instanceId);
return new Response(response.body, { status: response.status, statusText: response.statusText, headers });
}
}

/**
* The server's last step for every answered request: `GET /v1/models` names the serving Home,
* whichever branch produced the response. `pathname` is the dispatch path the router matched.
*/
export function stampServedBy(method: string, pathname: string, response: Response): Response {
if (!response || method !== "GET" || pathname !== "/v1/models") return response;
return withServedBy(response);
}
11 changes: 10 additions & 1 deletion src/server/index/serve-options.ts
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,7 @@ import type { ReadinessGate } from "../readiness";
import type { WorkflowRefusalLog } from "../workflow-refusal";

import { readyProtocolMetadata } from "../../remote/protocol";
import { homeInstanceId, stampServedBy } from "../../remote/served-by";
import { modelCapabilityFields } from "../models-capabilities";
import { createWebsocketHandler } from "./websocket-handler";
import { reasoningArchiveErrorResponse } from "../responses/reasoning-archive-error";
Expand Down Expand Up @@ -413,11 +414,13 @@ export function createServeOptions(ctx: ServeOptionsContext) {
const message = "OpenCodex package files changed while this proxy was running; it restarts on its own, or run 'ocx restart' ('ocx service restart' for a background service).";
const fencedPort = ctx.boundPort ?? requestServer.port ?? listenPort;
const fencedHealth = url.pathname === "/healthz";
const fencedInstanceId = fencedHealth ? homeInstanceId() : undefined;
const response = fencedHealth || readyzPath !== undefined
? jsonResponse({
status: "restart_required",
service: "opencodex",
version: VERSION,
...(fencedInstanceId ? { instanceId: fencedInstanceId } : {}),
uptime: process.uptime(),
pid: process.pid,
port: fencedPort,
Expand Down Expand Up @@ -628,10 +631,13 @@ export function createServeOptions(ctx: ServeOptionsContext) {
if (url.pathname === "/healthz" && req.method === "GET") {
// service/pid/port let CLI liveness reject foreign 200s and verify pid identity.
const healthPort = ctx.server.port ?? listenPort;
// Opaque Home instance id (src/remote/served-by.ts); omitted until the identity exists.
const instanceId = homeInstanceId();
const response = jsonResponse({
status: "ok",
service: "opencodex",
version: VERSION,
...(instanceId ? { instanceId } : {}),
Comment thread
lidge-jun marked this conversation as resolved.
Resolved
uptime: process.uptime(),
pid: process.pid,
port: healthPort,
Expand Down Expand Up @@ -1982,5 +1988,8 @@ export function createServeOptions(ctx: ServeOptionsContext) {
},
websocket: createWebsocketHandler(ctx, requestMetrics),
} as const;
return serveOptions;
const routeFetch = serveOptions.fetch;
// x-opencodex-served-by on GET /v1/models, whichever branch answered (src/remote/served-by.ts).
return { ...serveOptions, fetch: async (req: Request, requestServer: Server<WsData>): Promise<Response> =>
stampServedBy(req.method, codexCompatibleUrl(req.url).pathname, await routeFetch(req, requestServer)) } as const;
}
8 changes: 7 additions & 1 deletion structure/quota-pool.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,16 @@ in-place exclusive create where hard links are unavailable), so two servers star
home adopt the same identity, and a symlink or malformed file at the path is refused and left
untouched rather than overwritten.

`homeId` is also the Home instance id other peers see: `GET /healthz` publishes it as `instanceId`
and `GET /v1/models` names it in `x-opencodex-served-by` (`src/remote/served-by.ts`, see
[Runtime](runtime.md)). It is random and carries nothing derivable from the machine; the secret is
never published.

`accountRef` is `acct_` + base64url of the first 128 bits of HMAC-SHA256(secret,
`ocx.quota-pool/1` NUL provider NUL accountId). It is stable across restarts, distinct per provider,
and never contains an email, a raw account id or a token; without the secret a recipient cannot
confirm a guessed identity against it. Deleting the file rotates `homeId` and every reference
together. `label` is the account's existing log label (`main`, `p<hex6>`, `o<hex6>`), never an
together when the proxy next starts; a running proxy keeps the identity it already read (and keeps
publishing that `homeId` as its instance id). `label` is the account's existing log label (`main`, `p<hex6>`, `o<hex6>`), never an
email or alias. `producer.role` is `runtimeRole` (`standalone` when unset) and `producer.version`
is the package version. `driver` and `pooled.headroom` are not produced yet.
2 changes: 2 additions & 0 deletions structure/remote-link.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ Enrollment cancellation in `src/client/connect.ts` reaches actual network reques
The production relay's complete-send boundary is `src/client/link-relay-transport.ts`. A dedicated HTTP agent opens exactly one TCP socket. Before any admission header or body is sent, a credential-free, five-second challenge proves the stored link/key identity using direction-separated HMACs from `src/link/relay-auth.ts`. The data request uses that same authenticated socket, with a one-use session nonce; a closed socket cannot be silently replaced. A peer lacking this protocol is refused with an upgrade/re-link error, never retried through an unauthenticated send. Home-initiated reverse tunnels use the same check. `LinkRelayDeps.fetchImpl` is a trusted complete-send test seam, not a user setting; production supplies no substitute. Bun's Node HTTP compatibility is confined to this leaf so streamed request caps, backpressure, credential replacement, decompression, response status, caller cancellation and SSE limits remain owned by the existing relay.
A Home advertises `LINK_RELAY_AUTH_VERSION` as `relay-auth/<n>`, and the management relay as `link-management-relay`, in its `/readyz` `features` ([Runtime](runtime.md), `src/remote/features.ts`), so a peer can detect a relay-auth mismatch without comparing release versions.

The relay copies the Home's response headers except `content-encoding`, `content-length` and hop-by-hop headers, so the Home's `x-opencodex-served-by` on `GET /v1/models` reaches the Child's caller unchanged and the Child never stamps its own; `tests/server/served-by-header.test.ts` relays through the Home's real link listener and through a fake Home naming another id ([Runtime](runtime.md)).

`src/server/index/link-relay-sessions.ts` owns a per-listener reservation book. A valid caller proof reserves its peer socket for five seconds (at most 256 outstanding reservations); the data request must consume that nonce on the same peer address/port while its link and credential remain valid. Current and unexpired pending-rotation fingerprints use the same expiry rule as ordinary API-key admission, and the server signs with the fingerprint that matched. The nonce is not an alternative data-plane credential: dispatch still applies its ordinary key and route policy. Removal, rotation expiry, abort and commit are observed anew at dispatch. Replays and a different socket are refused.

Last-link `close()` refuses new challenges/work, retains both the bound port and hub-link ingress identity through pending reservations and active response bodies, then closes. An abandoned proof expires; EOF, error and cancellation release an active response. Process `stop()` remains forceful: it retires reservations and closes every socket, which the client cannot reconnect. The regression file `tests/clients/link-relay-bound-transport.test.ts` exercises the actual sockets, shutdown, same-connection proof, refusal to reconnect, bounded reservations, current/pending key lookup and response decoding. This is port-replacement protection, not a claim of encryption against an active transparent local intermediary. SSH continues to own remote-host authentication/encryption; enrollment and supervisor probes are separate boundaries.
Expand Down
2 changes: 1 addition & 1 deletion structure/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ config snapshot and the bounded service-token observation needed for its `_remot
it does not import the connect command, inspect catalog readiness, acquire lifecycle locks, or run
config/secret ACL hardening.

`src/remote/protocol.ts` owns pure interval/feature negotiation. `src/remote/features.ts` is the one registry of peer features `GET /readyz` lists in `features`, and each entry reads its owning source: `quota-pool/1` when `src/server/management/route-registry.ts` declares `GET /api/quota-pool`, `link-management-relay` when `src/link/management-routes.ts` has read paths, `served-by-header` when the emitter descriptor `src/remote/served-by.ts` says responses carry `x-opencodex-served-by`, and `relay-auth/<n>` from `src/link/relay-auth.ts`. The ids are fixed strings with no account or configuration data; `tests/server/readyz-features.test.ts` checks presence and absence per source, the peer parser, and the served-by advertisement against live responses. The Child machine listener answers its own `/readyz` without `features`. `src/remote/hub-state.ts` owns the `GET|HEAD /v1/hub-state` contract, its caps, and the parser both sides share. `src/client/hub-client.ts` owns bounded, schema-validated remote catalog consumption, hub-state reads, and key-id probes; `src/client/hub-state.ts` owns the resolution and the owner-stamped 0600 cache, and a failed read reports "unavailable" rather than degrading to the client's own local provider and login state. `src/client/hub-relay.ts` is a fixed-authority management relay with URL, header, body, redirect, and stream bounds. The public data listener remains the direct client→hub path; the loopback management ingress never serves data-plane routes. A client with `transport: "link"` reaches its hub through an SSH tunnel instead; see [Remote Link](remote-link.md). A client with a link sidecar owns its SSH tunnel; see [Remote Link](remote-link.md). For Remote Link, `src/cli/status.ts` asks the identity-checked local Child for `/v1/hub-state` without any credential header; the Child's connection-bound relay owns Home authentication. Hub Homes retain their bounded parsed projection and owner-bound cache. `src/client/hub-client.ts` distinguishes a bounded nested `hub_state_not_a_hub` 404 from legacy route absence; `src/client/hub-state.ts` reports role absence without presenting obsolete cached Hub state. `src/client/link-status.ts` reads matching sidecar/marker evidence for initiation identity; ambiguous data does not claim a Home-owned tunnel. These contracts are covered in `tests/cli/cli-status-hub-state.test.ts` and `tests/clients/link-status-projection.test.ts`.
`src/remote/protocol.ts` owns pure interval/feature negotiation. `src/remote/features.ts` is the one registry of peer features `GET /readyz` lists in `features`, and each entry reads its owning source: `quota-pool/1` when `src/server/management/route-registry.ts` declares `GET /api/quota-pool`, `link-management-relay` when `src/link/management-routes.ts` has read paths, `served-by-header` when the emitter descriptor `src/remote/served-by.ts` says responses carry `x-opencodex-served-by`, and `relay-auth/<n>` from `src/link/relay-auth.ts`. `src/remote/served-by.ts` also owns the value: the Home instance id is the install's `homeId` from `src/quota-pool/identity.ts` (random, persisted, never a hostname), published on `GET /healthz` (including its package-tree-fenced 503 body) as `instanceId` and stamped on every `GET /v1/models` response by the wrapper at the end of `createServeOptions` (`src/server/index/serve-options.ts`), whichever branch answered. While the identity is unavailable both are omitted rather than sent empty. A link Child's relay passes the Home's header through and a Hub-mode client's Codex reads the Hub data listener directly, so the header always names the serving Home; `tests/server/served-by-header.test.ts` covers standalone, Hub, link-relayed, restart-stable, opaque and unavailable-identity cases. The ids are fixed strings with no account or configuration data; `tests/server/readyz-features.test.ts` checks presence and absence per source, the peer parser, and the served-by advertisement against live responses. The Child machine listener answers its own `/readyz` without `features`. `src/remote/hub-state.ts` owns the `GET|HEAD /v1/hub-state` contract, its caps, and the parser both sides share. `src/client/hub-client.ts` owns bounded, schema-validated remote catalog consumption, hub-state reads, and key-id probes; `src/client/hub-state.ts` owns the resolution and the owner-stamped 0600 cache, and a failed read reports "unavailable" rather than degrading to the client's own local provider and login state. `src/client/hub-relay.ts` is a fixed-authority management relay with URL, header, body, redirect, and stream bounds. The public data listener remains the direct client→hub path; the loopback management ingress never serves data-plane routes. A client with `transport: "link"` reaches its hub through an SSH tunnel instead; see [Remote Link](remote-link.md). A client with a link sidecar owns its SSH tunnel; see [Remote Link](remote-link.md). For Remote Link, `src/cli/status.ts` asks the identity-checked local Child for `/v1/hub-state` without any credential header; the Child's connection-bound relay owns Home authentication. Hub Homes retain their bounded parsed projection and owner-bound cache. `src/client/hub-client.ts` distinguishes a bounded nested `hub_state_not_a_hub` 404 from legacy route absence; `src/client/hub-state.ts` reports role absence without presenting obsolete cached Hub state. `src/client/link-status.ts` reads matching sidecar/marker evidence for initiation identity; ambiguous data does not claim a Home-owned tunnel. These contracts are covered in `tests/cli/cli-status-hub-state.test.ts` and `tests/clients/link-status-projection.test.ts`.

### Remote Hub status credential binding

Expand Down
Loading
Loading