From dd11937b11709cf71c2b03ca36b2450bf83a28c1 Mon Sep 17 00:00:00 2001 From: pnguyen44 Date: Wed, 9 Sep 2026 13:14:03 -0400 Subject: [PATCH 1/6] HYPERFLEET-1519 - docs: SPIKE remote applier Postgres connectivity and partition isolation --- .../spike-remote-applier-postgres-access.md | 158 ++++++++++++++++++ 1 file changed, 158 insertions(+) create mode 100644 hyperfleet/docs/spike-remote-applier-postgres-access.md diff --git a/hyperfleet/docs/spike-remote-applier-postgres-access.md b/hyperfleet/docs/spike-remote-applier-postgres-access.md new file mode 100644 index 00000000..aab4ddba --- /dev/null +++ b/hyperfleet/docs/spike-remote-applier-postgres-access.md @@ -0,0 +1,158 @@ +--- +Status: Active +Owner: HyperFleet Team +Last Updated: 2026-09-09 +--- + +# SPIKE: Remote Applier Connectivity and Partition-Scoped Access to Postgres + +**Jira**: [HYPERFLEET-1519](https://redhat.atlassian.net/browse/HYPERFLEET-1519) + +**Parent epic**: [HYPERFLEET-1518](https://redhat.atlassian.net/browse/HYPERFLEET-1518) (Desire Store on Postgres) + +**Prior art**: + +- [Desire store on API Postgres spike (HYPERFLEET-1432)](desire-store-api-postgres-spike.md) +- [Desire identity and ownership spike (HYPERFLEET-1421)](spike-desire-identity-ownership.md) + +## Table of Contents + +- [Overview](#overview) +- [API-mediated vs. direct DB access](#api-mediated-vs-direct-db-access) +- [Recommended: API-mediated access](#recommended-api-mediated-access) +- [Alternative considered: Direct DB access](#alternative-considered-direct-db-access) + - [Network exposure](#network-exposure) + - [Authentication](#authentication) + - [Partition scoping](#partition-scoping) +- [Latency considerations](#latency-considerations) +- [Decision](#decision) +- [Trade-offs](#trade-offs) +- [Open questions](#open-questions) + +--- + +## Overview + +The Postgres desire store spike ([HYPERFLEET-1432](https://redhat.atlassian.net/browse/HYPERFLEET-1432)) validated Postgres as the desire store backend. The desire identity spike ([HYPERFLEET-1421](https://redhat.atlassian.net/browse/HYPERFLEET-1421)) deferred cryptographic enforcement to production backends. This spike resolves two remaining questions: + +- **Connectivity**: how do the adapter and applier, running on separate OCI clusters, connect to the desire store? +- **Partition isolation**: how do we ensure applier-A can only access cluster-A's rows? + +Deployment is OCI-only. The hub (API, Postgres, Sentinel, Broker) and management clusters (HyperShift, Applier) both run on OCI, on separate clusters; on-prem/multi-cloud portability is a future scenario (ADR-0019), not a requirement here. + +--- + +## API-mediated vs. direct DB access + +The central question: should clients connect to the desire store through an API layer, or directly to Postgres? + +| Dimension | API-mediated access | Direct DB access | +|-----------|---------------------|-------------------| +| Postgres exposure | Never exposed | Exposed via network path (internal or external, depending on mechanism) | +| Partition isolation | Application-level, in the API layer | Requires RLS, per-cluster credentials, or app-level enforcement at the client | +| Credential lifecycle | None; JWT via K8s TokenRequest API, auto-rotated | Requires dedicated design; complexity depends on mechanism chosen | +| New components to build | New endpoints (existing API or separate service) | None, if using existing cloud/proxy infra; a service, if credential-brokering is added | +| Single point of failure | Yes, if endpoints are added to the existing API | No new single point, beyond Postgres itself | + +--- + +## Recommended: API-mediated access + +Don't expose Postgres directly. Put a gRPC/REST service on the hub in front of the desire store. Both the adapter and applier talk to the service, not to the DB. + +- Pros: + - Postgres is never exposed + - Access control enforced in application code + - Single endpoint to secure + - Partition isolation becomes an application-level problem, simpler to reason about and test + - Decouples adapter and applier from DB backend; could swap Postgres for another store without changing clients + - May be able to reuse existing Envoy + Authorino auth infrastructure +- Cons: + - New endpoints to build (on existing API or as a separate service) + - Adds a network hop and serialization overhead + - Desire store interface must be reimplemented as REST/gRPC endpoints + - API becomes a single point of failure for both resource CRUD and desire delivery (if endpoints are added to existing API) + +**Partition isolation**: enforced in the API layer's application code. No RLS, per-cluster DB credentials, or credential-brokering service needed on the DB side. + +**Authentication**: clients authenticate via JWT through Envoy + Authorino (same as Sentinel today). + +**Credential lifecycle**: handled by existing infrastructure. Tokens are short-lived and auto-rotated via the K8s TokenRequest API. No manual provisioning or revocation workflow needed. Decommissioning a management cluster means deleting its service account; tokens become invalid immediately. + +--- + +## Alternative considered: Direct DB access + +Clients connect to Postgres directly without an API layer. This requires three additional decisions, each with operational overhead that API-mediated access avoids, since Postgres is never exposed to the adapter or applier. + +### Network exposure + +How a client reaches Postgres from a different cluster. + +- **VPN / tunnel**: works everywhere, Postgres stays internal; but requires VPN infrastructure HyperFleet doesn't have, plus operational overhead (key management, monitoring, failover) +- **External endpoint with mTLS**: simple topology, no VPN infra; but exposes Postgres publicly (even if mTLS-gated), larger attack surface, certificate distribution and rotation needed +- **Proxy sidecar** (e.g. PgBouncer): client code is unaware of connectivity details, good for connection pooling at scale; but still needs an underlying tunnel or endpoint, extra process to deploy per client +- **Cloud-native private link** (OCI Service Gateway): low latency, cloud-managed, no public IP; but cloud-specific setup, doesn't cover on-prem or air-gapped, requires a cloud-managed DB service + +### Authentication + +How a client proves its identity to Postgres. + +- **Username/password over TLS**: works everywhere, simple to implement; but credentials are static secrets that must be distributed and rotated +- **mTLS client certificates**: no shared secret to leak, identity is cryptographic; but certificate provisioning and rotation needed per client, not all cloud-managed Postgres services support it +- **Cloud IAM auth** (e.g. OCI instance principal): no static credentials, short-lived tokens auto-rotated; but cloud-specific, doesn't work on-prem or cross-cloud, requires a cloud-managed DB service + +### Partition scoping + +How a client is restricted to its own management cluster's rows. + +- **Application-level enforcement**: simple to implement, easy to test; but a bug in the store library could leak rows across partitions, no defense-in-depth +- **RLS with shared credentials**: DB enforces isolation, even a buggy client can't read another partition's rows; but client must set session variable correctly, harder to debug (silent empty results if wrong) +- **Per-cluster DB credentials**: strongest isolation (credential = identity = partition scope); but credential provisioning scales linearly with clusters, rotation and revocation complexity increases per cluster +- **Credential-brokering service**: short-lived credentials reduce blast radius; but new service to build and operate, becomes a single point of failure, still needs RLS or per-cluster credentials at the DB layer + +--- + +## Latency considerations + +The poll interval is 5s (assumed from [HYPERFLEET-1432](https://redhat.atlassian.net/browse/HYPERFLEET-1432) load modeling, configurable). Each poll does a partition read and status writes. This applies regardless of which path (API-mediated or direct DB) is chosen. + +Clients connect from separate OCI clusters. Same-cloud, same-region round-trip latency is expected to be low. The 5s poll interval provides a large budget; the extra network hop from API-mediated access is not a concern. + +--- + +## Decision + +**API-mediated access**, for the reasons above: it reuses existing auth infrastructure, avoids three additional decision dimensions that direct DB access requires (network exposure, authentication, partition scoping), and the 5s poll interval makes the extra network hop negligible. See [Trade-offs](#trade-offs) below for what this costs. + +--- + +## Trade-offs + +### What We Gain + +- Postgres is never exposed outside the hub cluster; no DB-level network exposure, authentication, or partition-scoping decisions to make +- Partition isolation is application code, testable with unit tests, no RLS or per-cluster DB credentials +- Credential lifecycle is already solved: JWT via K8s TokenRequest API, auto-rotated, revocable by deleting a service account +- Reuses Envoy + Authorino auth infrastructure already planned for the hub +- Adapter and applier use the same auth path as Sentinel, reducing the number of auth mechanisms in the system + +### What We Lose / What Gets Harder + +- New endpoints must be built for the desire store interface (partition read, status write, desire write) +- The API becomes a single point of failure for both resource CRUD and desire delivery (if endpoints are added to the existing API); separating into a dedicated service avoids this but adds deployment complexity +- Adds a network hop and serialization overhead between client and store +- Tighter coupling between client release cadence and API release cadence (clients depend on API contract stability) + +### Acceptable Because + +- The desire store interface is small and well-defined (partition read, status write, desire write); the endpoint surface area is bounded +- The 5s poll interval makes the extra network hop negligible +- API-mediated access eliminates three entire categories of decisions (network exposure, authentication, partition scoping) and their associated operational overhead +- The API is already a hard dependency for the control plane (Sentinel already depends on it for cluster/nodepool state); routing desire store traffic through it adds load to an already-critical path, not a new failure mode + +--- + +## Open questions + +- **OCI security constraints**: Ask the Oracle team about OCI-specific security requirements or restrictions (e.g. network policies, required auth mechanisms, managed DB limitations). With API-mediated access, this is no longer blocking the path decision, but may still be relevant for the hub cluster's internal Postgres setup. From e9fd2d6176a9c79fe091bd1ad1c67a977639ffa7 Mon Sep 17 00:00:00 2001 From: pnguyen44 Date: Wed, 9 Sep 2026 13:50:29 -0400 Subject: [PATCH 2/6] HYPERFLEET-1519 - docs: refine remote applier spike --- .../docs/spike-remote-applier-postgres-access.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/hyperfleet/docs/spike-remote-applier-postgres-access.md b/hyperfleet/docs/spike-remote-applier-postgres-access.md index aab4ddba..7366662b 100644 --- a/hyperfleet/docs/spike-remote-applier-postgres-access.md +++ b/hyperfleet/docs/spike-remote-applier-postgres-access.md @@ -73,11 +73,11 @@ Don't expose Postgres directly. Put a gRPC/REST service on the hub in front of t - Desire store interface must be reimplemented as REST/gRPC endpoints - API becomes a single point of failure for both resource CRUD and desire delivery (if endpoints are added to existing API) -**Partition isolation**: enforced in the API layer's application code. No RLS, per-cluster DB credentials, or credential-brokering service needed on the DB side. +**Partition isolation**: enforced in the API layer. Partition scope is derived from the caller's verified service identity (JWT via Envoy + Authorino), not from any client-supplied partition parameter. Mismatched or override attempts are rejected. No RLS, per-cluster DB credentials, or credential-brokering service needed on the DB side. -**Authentication**: clients authenticate via JWT through Envoy + Authorino (same as Sentinel today). +**Client authentication** (adapter/applier → API): JWT through Envoy + Authorino (same as Sentinel today). -**Credential lifecycle**: handled by existing infrastructure. Tokens are short-lived and auto-rotated via the K8s TokenRequest API. No manual provisioning or revocation workflow needed. Decommissioning a management cluster means deleting its service account; tokens become invalid immediately. +**Client credential lifecycle**: handled by existing infrastructure. Tokens are short-lived and auto-rotated via the K8s TokenRequest API. No manual provisioning or rotation workflow needed. Decommissioning a management cluster means deleting its service account; bound tokens then fail TokenReview after a short invalidation window (Kubernetes typically allows up to ~60s after `metadata.deletionTimestamp`), and any remaining lifetime ends at token expiry. --- @@ -117,13 +117,13 @@ How a client is restricted to its own management cluster's rows. The poll interval is 5s (assumed from [HYPERFLEET-1432](https://redhat.atlassian.net/browse/HYPERFLEET-1432) load modeling, configurable). Each poll does a partition read and status writes. This applies regardless of which path (API-mediated or direct DB) is chosen. -Clients connect from separate OCI clusters. Same-cloud, same-region round-trip latency is expected to be low. The 5s poll interval provides a large budget; the extra network hop from API-mediated access is not a concern. +Clients connect from separate OCI clusters. Same-cloud, same-region round-trip latency is expected to be low. Against a 5s poll interval, the extra network hop from API-mediated access is expected to fit the budget; this should be confirmed with load testing during implementation if production concurrency is a concern. --- ## Decision -**API-mediated access**, for the reasons above: it reuses existing auth infrastructure, avoids three additional decision dimensions that direct DB access requires (network exposure, authentication, partition scoping), and the 5s poll interval makes the extra network hop negligible. See [Trade-offs](#trade-offs) below for what this costs. +**API-mediated access**, for the reasons above: it reuses existing auth infrastructure, avoids three additional decision dimensions that direct DB access requires (network exposure, authentication, partition scoping), and the 5s poll interval is expected to leave headroom for the extra network hop. See [Trade-offs](#trade-offs) below for what this costs. --- @@ -133,7 +133,7 @@ Clients connect from separate OCI clusters. Same-cloud, same-region round-trip l - Postgres is never exposed outside the hub cluster; no DB-level network exposure, authentication, or partition-scoping decisions to make - Partition isolation is application code, testable with unit tests, no RLS or per-cluster DB credentials -- Credential lifecycle is already solved: JWT via K8s TokenRequest API, auto-rotated, revocable by deleting a service account +- Client credential lifecycle is already solved: JWT via K8s TokenRequest API, auto-rotated, revocable by deleting a service account (short invalidation window, then token expiry) - Reuses Envoy + Authorino auth infrastructure already planned for the hub - Adapter and applier use the same auth path as Sentinel, reducing the number of auth mechanisms in the system @@ -147,9 +147,9 @@ Clients connect from separate OCI clusters. Same-cloud, same-region round-trip l ### Acceptable Because - The desire store interface is small and well-defined (partition read, status write, desire write); the endpoint surface area is bounded -- The 5s poll interval makes the extra network hop negligible +- The 5s poll interval is expected to leave headroom for the extra network hop - API-mediated access eliminates three entire categories of decisions (network exposure, authentication, partition scoping) and their associated operational overhead -- The API is already a hard dependency for the control plane (Sentinel already depends on it for cluster/nodepool state); routing desire store traffic through it adds load to an already-critical path, not a new failure mode +- Remote appliers gain an API dependency for partition reads and status writes (unlike direct DB access, which would depend on Postgres availability instead). That still lands on the same hub failure domain Sentinel already relies on for cluster/nodepool state; client timeout/retry/degraded behavior is left to implementation --- From 30b160dfcfdc2ced6dd15fc9212123caf5098d22 Mon Sep 17 00:00:00 2001 From: pnguyen44 Date: Wed, 9 Sep 2026 14:06:25 -0400 Subject: [PATCH 3/6] HYPERFLEET-1519 - docs: clarify partition scope comes from Authorino-injected headers --- hyperfleet/docs/spike-remote-applier-postgres-access.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hyperfleet/docs/spike-remote-applier-postgres-access.md b/hyperfleet/docs/spike-remote-applier-postgres-access.md index 7366662b..a6ad09ea 100644 --- a/hyperfleet/docs/spike-remote-applier-postgres-access.md +++ b/hyperfleet/docs/spike-remote-applier-postgres-access.md @@ -73,7 +73,7 @@ Don't expose Postgres directly. Put a gRPC/REST service on the hub in front of t - Desire store interface must be reimplemented as REST/gRPC endpoints - API becomes a single point of failure for both resource CRUD and desire delivery (if endpoints are added to existing API) -**Partition isolation**: enforced in the API layer. Partition scope is derived from the caller's verified service identity (JWT via Envoy + Authorino), not from any client-supplied partition parameter. Mismatched or override attempts are rejected. No RLS, per-cluster DB credentials, or credential-brokering service needed on the DB side. +**Partition isolation**: enforced in the API layer. Envoy strips caller-supplied identity/tenant headers; Authorino validates the JWT and injects trusted identity headers. The API derives partition scope only from those injected headers (not from JWT claims or any client-supplied partition parameter). Mismatched or override attempts are rejected. No RLS, per-cluster DB credentials, or credential-brokering service needed on the DB side. **Client authentication** (adapter/applier → API): JWT through Envoy + Authorino (same as Sentinel today). From ce82f8e9d8808819d808cd9fead05170d4a5bac7 Mon Sep 17 00:00:00 2001 From: pnguyen44 Date: Thu, 10 Sep 2026 12:04:00 -0400 Subject: [PATCH 4/6] HYPERFLEET-1519 - docs: record API-mediated desire store access decision --- .../0022-api-mediated-desire-store-access.md | 52 +++++++ hyperfleet/adrs/README.md | 1 + .../spike-remote-applier-postgres-access.md | 146 +++--------------- 3 files changed, 76 insertions(+), 123 deletions(-) create mode 100644 hyperfleet/adrs/0022-api-mediated-desire-store-access.md diff --git a/hyperfleet/adrs/0022-api-mediated-desire-store-access.md b/hyperfleet/adrs/0022-api-mediated-desire-store-access.md new file mode 100644 index 00000000..b81dc872 --- /dev/null +++ b/hyperfleet/adrs/0022-api-mediated-desire-store-access.md @@ -0,0 +1,52 @@ +--- +Status: Active +Owner: HyperFleet Architecture Team +Last Updated: 2026-09-10 +--- + +# 0022 - API-Mediated Desire Store Access + +## Context + +The Postgres desire store is hosted on the hub cluster, while each Applier runs on a separate OCI management cluster. The Adapter runs on the hub and is co-located with Postgres. Remote Appliers must read their partition's desires and write desire status without gaining access to other management clusters' rows. + +Direct Postgres access would require separate decisions for cross-cluster network exposure, client credential lifecycle, and partition enforcement. It would also expose the database beyond the hub cluster. HyperFleet already uses the Envoy and Authorino gateway to authenticate internal callers and inject trusted identity headers. + +## Decision + +Adapters and Appliers access the desire store through an authenticated, hub-hosted API service. Postgres is not exposed directly to clients outside the hub cluster. + +The service exposes the bounded desire-store operations: partition reads, desire writes, and status writes. Envoy removes caller-supplied identity and tenant headers. The gateway validates each remote Applier through a trusted cross-cluster identity mechanism and maps the validated identity to exactly one partition before injecting trusted identity headers. The service derives partition scope exclusively from those injected headers, rejects requested scope that does not match the caller, and never derives scope from client-supplied parameters or untrusted JWT claims. + +The transport (REST or gRPC) and whether the endpoints run in the existing API or a dedicated hub service are implementation decisions. Both options must use the gateway and partition-scoping contract defined here. + +## Consequences + +**Gains:** + +- Postgres remains inaccessible outside the hub cluster, eliminating direct DB network exposure and client database credentials. +- Partition isolation is enforced once in testable server-side application code, without Postgres row-level security or per-cluster database accounts. +- Clients depend on a stable desire-store contract rather than the Postgres schema, allowing the storage backend to change without client changes. + +**Trade-offs:** + +- The hub API service is on the availability path for desire reads and status writes. Client timeout, retry, and degraded-mode behavior must be defined by the implementation work. +- The service adds a network hop and serialization overhead. The assumed 5s poll interval is expected to accommodate this, but load testing must validate it at production concurrency. +- HyperFleet must implement and version the desire-store endpoints. Clients now have an API compatibility dependency rather than a database dependency. +- Hosting the endpoints in the existing API shares its failure domain. A dedicated service can isolate that risk but adds deployment complexity. +- The cross-cluster identity mechanism, including its issuer trust, credential lifecycle, revocation behavior, and Authorino configuration, requires a follow-up design. + +## Alternatives Considered + +| Alternative | Why Rejected | +|-------------|--------------| +| Direct Postgres access from Appliers | Requires cross-cluster database exposure, database credential distribution and rotation, and a separate partition-isolation design using row-level security, per-cluster credentials, or client enforcement. | +| Direct access through VPN, tunnel, private link, or proxy | Solves only connectivity. It still requires database authentication and partition-isolation mechanisms, and adds infrastructure or cloud-specific operational dependencies. | +| Per-cluster Postgres credentials | Provides strong database-level isolation but makes provisioning, rotation, and revocation scale with the number of management clusters. | +| Postgres row-level security with shared credentials | Adds database-level defense in depth but relies on correct client session state and does not remove the need to expose and authenticate direct database connections. | + +## References + +- [Remote Applier Connectivity and Partition-Scoped Access to Postgres spike](../docs/spike-remote-applier-postgres-access.md) +- [ADR-0020: Envoy and Authorino as the API Authentication Gateway](0020-envoy-authorino-api-gateway.md) +- [ADR-0019: Package HyperFleet as a Kubernetes Operator](0019-package-hyperfleet-as-operator.md) diff --git a/hyperfleet/adrs/README.md b/hyperfleet/adrs/README.md index 614ff9f4..0eaf96dd 100644 --- a/hyperfleet/adrs/README.md +++ b/hyperfleet/adrs/README.md @@ -97,3 +97,4 @@ What did we decide? State it plainly. | [0019](0019-package-hyperfleet-as-operator.md) | Package HyperFleet as a Kubernetes Operator | Proposed | 2026-08-11 | | [0020](0020-envoy-authorino-api-gateway.md) | Envoy and Authorino as the API Authentication Gateway | Active | 2026-08-11 | | [0021](0021-oci-external-platform.md) | External Platform for the Guest in OCI Hosted Clusters | Active | 2026-09-01 | +| [0022](0022-api-mediated-desire-store-access.md) | API-Mediated Desire Store Access | Active | 2026-09-10 | diff --git a/hyperfleet/docs/spike-remote-applier-postgres-access.md b/hyperfleet/docs/spike-remote-applier-postgres-access.md index a6ad09ea..806cb804 100644 --- a/hyperfleet/docs/spike-remote-applier-postgres-access.md +++ b/hyperfleet/docs/spike-remote-applier-postgres-access.md @@ -1,7 +1,7 @@ --- Status: Active Owner: HyperFleet Team -Last Updated: 2026-09-09 +Last Updated: 2026-09-10 --- # SPIKE: Remote Applier Connectivity and Partition-Scoped Access to Postgres @@ -18,141 +18,41 @@ Last Updated: 2026-09-09 ## Table of Contents - [Overview](#overview) -- [API-mediated vs. direct DB access](#api-mediated-vs-direct-db-access) -- [Recommended: API-mediated access](#recommended-api-mediated-access) -- [Alternative considered: Direct DB access](#alternative-considered-direct-db-access) - - [Network exposure](#network-exposure) - - [Authentication](#authentication) - - [Partition scoping](#partition-scoping) -- [Latency considerations](#latency-considerations) +- [Options Evaluated](#options-evaluated) +- [Recommendation](#recommendation) +- [Latency and Availability](#latency-and-availability) - [Decision](#decision) -- [Trade-offs](#trade-offs) -- [Open questions](#open-questions) - ---- +- [Open Questions](#open-questions) ## Overview -The Postgres desire store spike ([HYPERFLEET-1432](https://redhat.atlassian.net/browse/HYPERFLEET-1432)) validated Postgres as the desire store backend. The desire identity spike ([HYPERFLEET-1421](https://redhat.atlassian.net/browse/HYPERFLEET-1421)) deferred cryptographic enforcement to production backends. This spike resolves two remaining questions: - -- **Connectivity**: how do the adapter and applier, running on separate OCI clusters, connect to the desire store? -- **Partition isolation**: how do we ensure applier-A can only access cluster-A's rows? - -Deployment is OCI-only. The hub (API, Postgres, Sentinel, Broker) and management clusters (HyperShift, Applier) both run on OCI, on separate clusters; on-prem/multi-cloud portability is a future scenario (ADR-0019), not a requirement here. - ---- - -## API-mediated vs. direct DB access - -The central question: should clients connect to the desire store through an API layer, or directly to Postgres? - -| Dimension | API-mediated access | Direct DB access | -|-----------|---------------------|-------------------| -| Postgres exposure | Never exposed | Exposed via network path (internal or external, depending on mechanism) | -| Partition isolation | Application-level, in the API layer | Requires RLS, per-cluster credentials, or app-level enforcement at the client | -| Credential lifecycle | None; JWT via K8s TokenRequest API, auto-rotated | Requires dedicated design; complexity depends on mechanism chosen | -| New components to build | New endpoints (existing API or separate service) | None, if using existing cloud/proxy infra; a service, if credential-brokering is added | -| Single point of failure | Yes, if endpoints are added to the existing API | No new single point, beyond Postgres itself | - ---- - -## Recommended: API-mediated access - -Don't expose Postgres directly. Put a gRPC/REST service on the hub in front of the desire store. Both the adapter and applier talk to the service, not to the DB. - -- Pros: - - Postgres is never exposed - - Access control enforced in application code - - Single endpoint to secure - - Partition isolation becomes an application-level problem, simpler to reason about and test - - Decouples adapter and applier from DB backend; could swap Postgres for another store without changing clients - - May be able to reuse existing Envoy + Authorino auth infrastructure -- Cons: - - New endpoints to build (on existing API or as a separate service) - - Adds a network hop and serialization overhead - - Desire store interface must be reimplemented as REST/gRPC endpoints - - API becomes a single point of failure for both resource CRUD and desire delivery (if endpoints are added to existing API) - -**Partition isolation**: enforced in the API layer. Envoy strips caller-supplied identity/tenant headers; Authorino validates the JWT and injects trusted identity headers. The API derives partition scope only from those injected headers (not from JWT claims or any client-supplied partition parameter). Mismatched or override attempts are rejected. No RLS, per-cluster DB credentials, or credential-brokering service needed on the DB side. - -**Client authentication** (adapter/applier → API): JWT through Envoy + Authorino (same as Sentinel today). - -**Client credential lifecycle**: handled by existing infrastructure. Tokens are short-lived and auto-rotated via the K8s TokenRequest API. No manual provisioning or rotation workflow needed. Decommissioning a management cluster means deleting its service account; bound tokens then fail TokenReview after a short invalidation window (Kubernetes typically allows up to ~60s after `metadata.deletionTimestamp`), and any remaining lifetime ends at token expiry. - ---- - -## Alternative considered: Direct DB access - -Clients connect to Postgres directly without an API layer. This requires three additional decisions, each with operational overhead that API-mediated access avoids, since Postgres is never exposed to the adapter or applier. +The Postgres desire store runs on the hub cluster. The Adapter is co-located with it, while each Applier runs on a separate OCI management cluster. This spike evaluates how remote Appliers access the Postgres-backed desire store and how to prevent cross-partition access. OCI-only deployment is the current requirement; multi-cloud portability is a future concern covered by ADR-0019. -### Network exposure +## Options Evaluated -How a client reaches Postgres from a different cluster. +| Dimension | API-mediated access | Direct Postgres access | +|-----------|---------------------|------------------------| +| Postgres exposure | Remains inside the hub cluster | Requires a cross-cluster network path | +| Authentication | Requires a cross-cluster identity design through Envoy and Authorino | Requires database credentials, mTLS, cloud IAM, or a credential broker | +| Partition isolation | Enforced once in server-side application code | Requires row-level security, per-cluster credentials, or client enforcement | +| Client coupling | Stable service contract | Postgres schema and connectivity details | +| Availability | Hub API service is on the path | Postgres is directly on the path | -- **VPN / tunnel**: works everywhere, Postgres stays internal; but requires VPN infrastructure HyperFleet doesn't have, plus operational overhead (key management, monitoring, failover) -- **External endpoint with mTLS**: simple topology, no VPN infra; but exposes Postgres publicly (even if mTLS-gated), larger attack surface, certificate distribution and rotation needed -- **Proxy sidecar** (e.g. PgBouncer): client code is unaware of connectivity details, good for connection pooling at scale; but still needs an underlying tunnel or endpoint, extra process to deploy per client -- **Cloud-native private link** (OCI Service Gateway): low latency, cloud-managed, no public IP; but cloud-specific setup, doesn't cover on-prem or air-gapped, requires a cloud-managed DB service +## Recommendation -### Authentication +Use API-mediated access. It keeps Postgres inside the hub cluster and centralizes partition enforcement. [ADR-0022](../adrs/0022-api-mediated-desire-store-access.md) records the selected architecture, including the gateway authentication and partition-scoping requirements. -How a client proves its identity to Postgres. +## Latency and Availability -- **Username/password over TLS**: works everywhere, simple to implement; but credentials are static secrets that must be distributed and rotated -- **mTLS client certificates**: no shared secret to leak, identity is cryptographic; but certificate provisioning and rotation needed per client, not all cloud-managed Postgres services support it -- **Cloud IAM auth** (e.g. OCI instance principal): no static credentials, short-lived tokens auto-rotated; but cloud-specific, doesn't work on-prem or cross-cloud, requires a cloud-managed DB service +Appliers poll every 5s, based on the load-modeling assumption from [HYPERFLEET-1432](https://redhat.atlassian.net/browse/HYPERFLEET-1432). Same-region OCI latency is expected to leave sufficient headroom for the additional network hop, but implementation load testing must validate this at production concurrency. -### Partition scoping - -How a client is restricted to its own management cluster's rows. - -- **Application-level enforcement**: simple to implement, easy to test; but a bug in the store library could leak rows across partitions, no defense-in-depth -- **RLS with shared credentials**: DB enforces isolation, even a buggy client can't read another partition's rows; but client must set session variable correctly, harder to debug (silent empty results if wrong) -- **Per-cluster DB credentials**: strongest isolation (credential = identity = partition scope); but credential provisioning scales linearly with clusters, rotation and revocation complexity increases per cluster -- **Credential-brokering service**: short-lived credentials reduce blast radius; but new service to build and operate, becomes a single point of failure, still needs RLS or per-cluster credentials at the DB layer - ---- - -## Latency considerations - -The poll interval is 5s (assumed from [HYPERFLEET-1432](https://redhat.atlassian.net/browse/HYPERFLEET-1432) load modeling, configurable). Each poll does a partition read and status writes. This applies regardless of which path (API-mediated or direct DB) is chosen. - -Clients connect from separate OCI clusters. Same-cloud, same-region round-trip latency is expected to be low. Against a 5s poll interval, the extra network hop from API-mediated access is expected to fit the budget; this should be confirmed with load testing during implementation if production concurrency is a concern. - ---- +The hub API service becomes a dependency for desire reads and status writes. This is the same hub failure domain that Sentinel already depends on for cluster and NodePool state. Client timeout, retry, and degraded-mode behavior remain implementation work. ## Decision -**API-mediated access**, for the reasons above: it reuses existing auth infrastructure, avoids three additional decision dimensions that direct DB access requires (network exposure, authentication, partition scoping), and the 5s poll interval is expected to leave headroom for the extra network hop. See [Trade-offs](#trade-offs) below for what this costs. - ---- - -## Trade-offs - -### What We Gain - -- Postgres is never exposed outside the hub cluster; no DB-level network exposure, authentication, or partition-scoping decisions to make -- Partition isolation is application code, testable with unit tests, no RLS or per-cluster DB credentials -- Client credential lifecycle is already solved: JWT via K8s TokenRequest API, auto-rotated, revocable by deleting a service account (short invalidation window, then token expiry) -- Reuses Envoy + Authorino auth infrastructure already planned for the hub -- Adapter and applier use the same auth path as Sentinel, reducing the number of auth mechanisms in the system - -### What We Lose / What Gets Harder - -- New endpoints must be built for the desire store interface (partition read, status write, desire write) -- The API becomes a single point of failure for both resource CRUD and desire delivery (if endpoints are added to the existing API); separating into a dedicated service avoids this but adds deployment complexity -- Adds a network hop and serialization overhead between client and store -- Tighter coupling between client release cadence and API release cadence (clients depend on API contract stability) - -### Acceptable Because - -- The desire store interface is small and well-defined (partition read, status write, desire write); the endpoint surface area is bounded -- The 5s poll interval is expected to leave headroom for the extra network hop -- API-mediated access eliminates three entire categories of decisions (network exposure, authentication, partition scoping) and their associated operational overhead -- Remote appliers gain an API dependency for partition reads and status writes (unlike direct DB access, which would depend on Postgres availability instead). That still lands on the same hub failure domain Sentinel already relies on for cluster/nodepool state; client timeout/retry/degraded behavior is left to implementation - ---- +Adopt API-mediated desire-store access. The resulting architectural decision and its consequences are recorded in [ADR-0022](../adrs/0022-api-mediated-desire-store-access.md). -## Open questions +## Open Questions -- **OCI security constraints**: Ask the Oracle team about OCI-specific security requirements or restrictions (e.g. network policies, required auth mechanisms, managed DB limitations). With API-mediated access, this is no longer blocking the path decision, but may still be relevant for the hub cluster's internal Postgres setup. +- **OCI security constraints**: Confirm OCI-specific restrictions that affect the hub cluster's internal Postgres deployment. This does not block the API-mediated path. +- **Client failure behavior**: Define timeout, retry, and degraded-mode behavior when the API service is unavailable as part of the implementation work. From 864866cf1648bfc0cc72859c6746448af9b4e9a6 Mon Sep 17 00:00:00 2001 From: pnguyen44 Date: Thu, 10 Sep 2026 12:19:14 -0400 Subject: [PATCH 5/6] HYPERFLEET-1519 - docs: require fail-closed partition enforcement --- hyperfleet/adrs/0022-api-mediated-desire-store-access.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hyperfleet/adrs/0022-api-mediated-desire-store-access.md b/hyperfleet/adrs/0022-api-mediated-desire-store-access.md index b81dc872..7c74715f 100644 --- a/hyperfleet/adrs/0022-api-mediated-desire-store-access.md +++ b/hyperfleet/adrs/0022-api-mediated-desire-store-access.md @@ -16,7 +16,7 @@ Direct Postgres access would require separate decisions for cross-cluster networ Adapters and Appliers access the desire store through an authenticated, hub-hosted API service. Postgres is not exposed directly to clients outside the hub cluster. -The service exposes the bounded desire-store operations: partition reads, desire writes, and status writes. Envoy removes caller-supplied identity and tenant headers. The gateway validates each remote Applier through a trusted cross-cluster identity mechanism and maps the validated identity to exactly one partition before injecting trusted identity headers. The service derives partition scope exclusively from those injected headers, rejects requested scope that does not match the caller, and never derives scope from client-supplied parameters or untrusted JWT claims. +The service exposes the bounded desire-store operations: partition reads, desire writes, and status writes. Partition enforcement is mandatory and cannot be disabled by an optional server configuration flag. The service configuration must declare the partition dimension, and startup must fail if that dimension or its enforcement configuration is missing. Envoy removes caller-supplied identity and tenant headers. The gateway validates each remote Applier through a trusted cross-cluster identity mechanism and maps the validated identity to exactly one partition before injecting trusted identity headers. Every request must resolve a non-empty partition scope exclusively from those injected headers before reaching the data layer; missing or invalid tenant context is rejected, requested scope that does not match the caller is rejected, and no unscoped query is permitted. The service never derives scope from client-supplied parameters or untrusted JWT claims. The transport (REST or gRPC) and whether the endpoints run in the existing API or a dedicated hub service are implementation decisions. Both options must use the gateway and partition-scoping contract defined here. From 7b4246b4c3b0d5e47cb736ef77752c1897110262 Mon Sep 17 00:00:00 2001 From: pnguyen44 Date: Fri, 11 Sep 2026 15:11:43 -0400 Subject: [PATCH 6/6] HYPERFLEET-1519 - docs: clarify desire store design follow-ups --- hyperfleet/adrs/0022-api-mediated-desire-store-access.md | 7 +++++-- hyperfleet/docs/spike-remote-applier-postgres-access.md | 5 +++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/hyperfleet/adrs/0022-api-mediated-desire-store-access.md b/hyperfleet/adrs/0022-api-mediated-desire-store-access.md index 7c74715f..615375a1 100644 --- a/hyperfleet/adrs/0022-api-mediated-desire-store-access.md +++ b/hyperfleet/adrs/0022-api-mediated-desire-store-access.md @@ -1,7 +1,7 @@ --- Status: Active Owner: HyperFleet Architecture Team -Last Updated: 2026-09-10 +Last Updated: 2026-09-11 --- # 0022 - API-Mediated Desire Store Access @@ -18,7 +18,9 @@ Adapters and Appliers access the desire store through an authenticated, hub-host The service exposes the bounded desire-store operations: partition reads, desire writes, and status writes. Partition enforcement is mandatory and cannot be disabled by an optional server configuration flag. The service configuration must declare the partition dimension, and startup must fail if that dimension or its enforcement configuration is missing. Envoy removes caller-supplied identity and tenant headers. The gateway validates each remote Applier through a trusted cross-cluster identity mechanism and maps the validated identity to exactly one partition before injecting trusted identity headers. Every request must resolve a non-empty partition scope exclusively from those injected headers before reaching the data layer; missing or invalid tenant context is rejected, requested scope that does not match the caller is rejected, and no unscoped query is permitted. The service never derives scope from client-supplied parameters or untrusted JWT claims. -The transport (REST or gRPC) and whether the endpoints run in the existing API or a dedicated hub service are implementation decisions. Both options must use the gateway and partition-scoping contract defined here. +This extends ADR-0020's gateway caller model with a third caller type: remote, partition-scoped machines. The issuer trust and credential lifecycle for this caller type are follow-up design decisions in [HYPERFLEET-1645: Design the desire-store API service and cross-cluster Applier identity](https://redhat.atlassian.net/browse/HYPERFLEET-1645). + +The transport (REST or gRPC) and whether the endpoints run in the existing API or a dedicated hub service are follow-up design decisions, not implementation details. They must be recorded before implementation in [HYPERFLEET-1645: Design the desire-store API service and cross-cluster Applier identity](https://redhat.atlassian.net/browse/HYPERFLEET-1645). The design must evaluate end-to-end capacity across the gateway, service, and Postgres. Both options must use the gateway and partition-scoping contract defined here. ## Consequences @@ -35,6 +37,7 @@ The transport (REST or gRPC) and whether the endpoints run in the existing API o - HyperFleet must implement and version the desire-store endpoints. Clients now have an API compatibility dependency rather than a database dependency. - Hosting the endpoints in the existing API shares its failure domain. A dedicated service can isolate that risk but adds deployment complexity. - The cross-cluster identity mechanism, including its issuer trust, credential lifecycle, revocation behavior, and Authorino configuration, requires a follow-up design. +- Partition isolation resides in the service layer. A defect in partition-scope resolution could expose multiple partitions, and all remote Appliers share the hub service and Postgres capacity. Postgres row-level security and per-caller gateway quotas can be added as defense in depth without changing this decision; their evaluation is follow-up design work in [HYPERFLEET-1645: Design the desire-store API service and cross-cluster Applier identity](https://redhat.atlassian.net/browse/HYPERFLEET-1645). ## Alternatives Considered diff --git a/hyperfleet/docs/spike-remote-applier-postgres-access.md b/hyperfleet/docs/spike-remote-applier-postgres-access.md index 806cb804..7ab0239f 100644 --- a/hyperfleet/docs/spike-remote-applier-postgres-access.md +++ b/hyperfleet/docs/spike-remote-applier-postgres-access.md @@ -1,7 +1,7 @@ --- Status: Active Owner: HyperFleet Team -Last Updated: 2026-09-10 +Last Updated: 2026-09-11 --- # SPIKE: Remote Applier Connectivity and Partition-Scoped Access to Postgres @@ -54,5 +54,6 @@ Adopt API-mediated desire-store access. The resulting architectural decision and ## Open Questions +The remaining service-design questions, including cross-cluster identity, partition binding, endpoint semantics, availability behavior, hosting, transport, gateway capacity, and defense in depth, are tracked in [HYPERFLEET-1645: Design the desire-store API service and cross-cluster Applier identity](https://redhat.atlassian.net/browse/HYPERFLEET-1645). + - **OCI security constraints**: Confirm OCI-specific restrictions that affect the hub cluster's internal Postgres deployment. This does not block the API-mediated path. -- **Client failure behavior**: Define timeout, retry, and degraded-mode behavior when the API service is unavailable as part of the implementation work.