perf: reduce gateway launch time to under 15 seconds

[drew]

Problem

Gateway launch times are too slow. The current bootstrap and sandbox provisioning pipeline has multiple sequential bottlenecks that add unnecessary latency. We need to get the full gateway launch under 15 seconds, with sandbox spin-up taking ~5 seconds.

Current Architecture

The launch pipeline is heavily sequential:

1. Cluster bootstrap — k3s container start → wait for kubeconfig (up to 60s) → PKI reconciliation → wait for cluster health (up to 360s)
2. Gateway server startup — DB migrations → kube client init → TLS setup → background watchers → accept loop
3. Sandbox provisioning — CRD creation → pod scheduling → image pull → sandbox binary startup (policy fetch, provider env fetch, ephemeral CA generation, netns creation, proxy start, SSH server start, child process spawn)

Each of these stages has polling loops with generous timeouts and sequential operations that could be parallelized or eliminated.

Key Bottlenecks to Investigate

Cluster Bootstrap

• Sequential polling loops (kubeconfig wait, namespace wait, health check) with conservative intervals
• PKI reconciliation waits for namespace creation before proceeding
• Component images (navigator/server, navigator/sandbox) are pulled at runtime rather than bundled — first launch pays the full pull cost
• Health check uses start-period=20s before first probe, then polls every 5s

Gateway Server Startup

• DB migration runs synchronously before anything else
• Kube client initialization may be slow depending on kubeconfig discovery
• TLS cert loading is synchronous

Sandbox Spin-Up

• Multiple sequential gRPC calls (policy fetch, provider env fetch, inference route bundles)
• Ephemeral CA generation is a crypto operation that blocks startup
• Network namespace creation involves multiple ip command invocations
• Proxy, SSH server, and child process are started sequentially

Proposed Approach

Phase 1: Measure (instrument)

• Add structured timing spans to every stage of the launch pipeline
• Establish baseline measurements for each phase on both warm and cold starts
• Identify the actual top-N time sinks vs. assumed bottlenecks

Phase 2: Parallelize

• Overlap independent operations in sandbox startup (e.g., fetch policy + fetch provider env + generate ephemeral CA concurrently)
• Parallelize gateway server init where possible (DB connect, kube client, TLS loading)
• Reduce polling intervals and tighten timeouts for bootstrap readiness checks

Phase 3: Eliminate

• Pre-bundle or pre-cache component images in the cluster image to avoid runtime pulls
• Explore lazy initialization for non-critical subsystems (e.g., defer inference route bundle fetch)
• Consider pre-generating ephemeral CA material or caching it across sandbox restarts

Acceptance Criteria

• Gateway is fully operational (accepting connections) within 15 seconds of launch initiation
• Sandbox pods reach Ready state within ~5 seconds of CreateSandboxRequest (assuming images are cached)
• Instrumentation is in place to continuously track launch latency
• No regression in security posture (mTLS, sandbox isolation, policy enforcement)