Complete PKI support for multi-cluster QUIC transport

[sbaum1994]

Problem

Multi-cluster self-managed NVCF needs certificate-validated worker-to-control-
plane communication over QUIC. Load balancer and Gateway API termination do not
cover this path because the target deployments pass UDP through to the NVCF
request router.

The supported design uses cert-manager for service certificates, pluggable
issuers for managed and customer-owned PKI, and explicit public trust material
for compute-plane workers. Client identity remains PSAT/OIDC based. Mutual TLS
is not part of this epic.

Current status

The native hierarchy is 8 of 10 complete.

Completed:

• Configure the default self-managed PKI issuer #47 configures the default self-managed issuer contract.
• Issue TLS certificates for the LLM request router #48 issues request-router TLS certificates.
• Bootstrap transport trust in nvcf-cli #49 bootstraps transport trust through nvcf-cli.
• Inject transport trust into NVCA-managed containers #50 injects transport trust into NVCA-managed containers.
• Support customer-managed PKI issuers #51 supports external cert-manager and customer-managed issuers.
• Replace the BusyBox trust installer with nvca-trust-init #54 replaces the BusyBox trust installer.
• Validate LLM request-router advertised hostnames against certificate SANs #838 validates advertised hostnames against certificate SANs.
• fix(nvca): inject transport trust into Helm LLM utility pods #606 injects transport trust into Helm LLM utility pods.

Remaining native sub-issues:

• feat(self-managed): support a pre-created LLM request-router TLS Secret #947 adds a pre-created request-router TLS Secret mode. It must preserve
  customer ownership of issuance and renewal, avoid rendering a cert-manager
  Certificate, and keep the public worker trust bundle separate from the
  server private key.
• Validate self-managed PKI end to end #52 records released-artifact end-to-end validation. It must prove a real
  LLM workload connects over certificate-validated reverse QUIC and that wrong
  or missing trust fails closed.

Required related work outside the native hierarchy:

• PR feat(stargate): reload mounted TLS server identities without restarts #777 implements the server-identity reload portion of feat(stargate): hot reload transport TLS material #599. It must
  merge before this epic closes. Client trust reload remains separate in feat(stargate): hot reload client TLS trust bundles #931.

Integration status

PR #951 merged and closed #950. The self-managed stack now has one
dependency-stage owner for the nvcf-pki Helm release. Preserve that ownership
model when implementing #947.

PR #777 is open and mergeable, but still requires review and completion of its
current checks. Its previous live-cluster validation exercised the earlier
filesystem watcher. Before merge, rerun the rotation scenarios against a real
kubelet using the current 30-second polling implementation. Confirm activation
within the polling window, repeated rotations recover from transient rejected
reads, and the expiry gauge advances for every valid rotation.

End-to-end gap

Issue #52 has no implementation or recorded validation evidence. Complete it
after #947 and PR #777 with:

• An explicitly configured worker-reachable request-router endpoint.
• A single request-router replica. Multi-replica chart-owned routing is
  deferred in Expose the LLM request router to split-cluster workers #689.
• A hostname used consistently for the worker address, TLS SNI, and certificate
  SAN.
• Published charts and images rather than source-only render tests.
• Positive worker registration and invocation evidence.
• Wrong-root, wrong-hostname, missing-trust, and fingerprint-mismatch failures.
• Server certificate rotation without a request-router restart.
• Recovery after restoring valid trust material.

Deferred follow-up

These items do not block this epic, #52, or general availability:

• Support optional non-root transport trust installation #53: optional Restricted Pod Security hardening.
• Add gRPC RED metrics and tracing to the Stargate Kubernetes router #584: gRPC RED metrics and tracing.
• feat(stargate): hot reload client TLS trust bundles #931: client trust hot reload. Issue feat(stargate): hot reload transport TLS material #599 remains the parent for both reload
  halves, but its server-identity portion in PR feat(stargate): reload mounted TLS server identities without restarts #777 is required now.
• feat(stargate): reject unsupported server identities in secure mode #946: strict rejection of unsupported Stargate server identities.
• Expose the LLM request router to split-cluster workers #689: chart-owned multi-replica routing.
• Restrict OpenBao ingress in self-managed deployments #912: coordinated OpenBao network-policy hardening.

Completion criteria

• Merge PR fix(self-managed): declare the LLM PKI issuer in one Helmfile state #951 and close fix(self-managed): nvcf-pki release is declared in two Helmfile states #950.
• Validate the current PR feat(stargate): reload mounted TLS server identities without restarts #777 polling path against a real kubelet, clear
  review and checks, and merge the server-identity reload change.
• Complete and close feat(self-managed): support a pre-created LLM request-router TLS Secret #947.
• Complete and close Validate self-managed PKI end to end #52 with recorded released-artifact evidence.
• Reconcile feat(stargate): hot reload transport TLS material #599 after PR feat(stargate): reload mounted TLS server identities without restarts #777 merges. Keep the deferred client-trust work
  in feat(stargate): hot reload client TLS trust bundles #931 without treating the server-identity work as deferred.
• Confirm every remaining open item is either a native completion gate or
  explicitly recorded as follow-up.
• Close this epic only after the native hierarchy and this status agree.

[fr4nsyz]

Is this issue currently available for external contribution? I noticed the description mentions that work is already in progress, so I wanted to check before starting any implementation work. If there are remaining pieces that would be useful for a community contributor to take on, this seems right up my alley and I'd love to help out!

[athappa-nv]

Hi @fr4nsyz, thank you for reaching out and for your interest in contributing!

PKI is something that we're actively working on and targeting for the 0.7 release. At the moment we're driving the initial implementation internally. However, once the core work is in place, there may be follow-up areas where community contributions would be very valuable.

In the meantime, if there are other areas of the project you're interested in contributing to, let us know and we'd be happy to work together to identify issues that would be a good fit!