Skip to content

Add GlobalReplicationGroup resource support - #237

Open
backlog-burndown-bot wants to merge 24 commits into
aws-controllers-k8s:mainfrom
backlog-burndown-bot:feat/1048-global-replication-group
Open

backlog-burndown-bot wants to merge 24 commits into
aws-controllers-k8s:mainfrom
backlog-burndown-bot:feat/1048-global-replication-group

Conversation

@backlog-burndown-bot

@backlog-burndown-bot backlog-burndown-bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available: fixes aws-controllers-k8s/community#1048

Description of changes:

Adds the GlobalReplicationGroup resource (ElastiCache Global Datastore), and makes ReplicationGroup CRs work as datastore members. Supersedes #228; co-authored with @NicholasBlaskey, whose live testing established the API rules below.

GlobalReplicationGroup

  • ModifyGlobalReplicationGroup takes one logical change per request, so each reconcile applies one change (shard count via Increase/Decrease) and requeues until the delta is empty.
  • Fields a datastore inherits or AWS defaults (engine, version, node type, description, shard count, failover) produce no difference when the manifest leaves them unset. The read path still reports them, so adoption imports them.
  • Automatic failover has no top-level Describe field; it is derived from the members (hence ShowMemberInfo).
  • cacheParameterGroupName is an engine-upgrade input only: it is sent with an engine-family or major-version upgrade and never produces a delta.
  • Shard shrink keeps the lowest-ordered shard IDs. Needs your buy-in: users can't pick which shards go.
  • Tags go through the tagging APIs (util.SyncTags); create-time tags come from the SDK bump in chore: bump aws-sdk-go-v2 to v1.47.4 #247.
  • Requeues every 60s so status.members follows failovers and members joining or leaving. A CustomUpdateConditions hook keeps Synced tied to available/primary-only, as sagemaker-controller does.
  • primaryReplicationGroupRef/ID are create-time inputs. They are not resolved after the datastore exists, so a failover or an unsynced primary RG CR no longer blocks it, and they go stale after a failover. A new PRIMARY column shows the current primary. On adoption the ID is filled from the current PRIMARY member.
  • Delete disassociates every secondary first (each is kept as a standalone cluster) and retains the primary.

ReplicationGroup as a datastore member

AWS rejects changing node type, parameter group, engine, engine version, automatic failover, auto minor version upgrade, in-transit encryption, durability, shard count and primaryClusterID on a member, even to the current value (verified live on primary and secondary). While status.globalReplicationGroupInfo names a datastore:

  • differences on those fields are dropped from the delta, following rds-controller's DBInstance/DBCluster handling and Terraform's equivalent;
  • declared values stay in spec: nothing is copied from AWS, and update results keep the declared values, so GitOps tools don't fight the controller;
  • they are stripped from ModifyReplicationGroup, and the shard and failover paths are skipped;
  • an ACK.Advisory condition and the field docs say so. Once the group leaves, normal reconciliation resumes, so a stale node type or shard count fails with InvalidParameterCombination and a lower engine version is rejected until the spec is updated.

Membership changes happen outside the ReplicationGroup, so every ReplicationGroup now requeues every 600s; --reconcile-resource-resync-seconds still overrides this per kind. The status-based Synced condition customSetOutput already sets is unaffected.

Testing

  • e2e covers create, modify (one shared datastore to limit provisioning time), tags, and delete.
  • Unit tests only: shard scaling (slow), the major-upgrade-plus-parameter-group path (not validated live), member handling and failover (need a cross-region setup CI lacks).

Known limitations

  • The adopt-or-create adoption policy needs primaryReplicationGroupRef or ID.
  • status.authTokenEnabled stays false after an auth token is added to a member, because that is what AWS reports.

Excluded (pre-existing, separate follow-ups)

  • util.EngineVersionsMatch panics on an empty version; guarded locally here.
  • ReplicationGroup's setNodeGroupConfiguration never populates the read-back slice, so a declared nodeGroupConfiguration always shows a difference.
  • Standalone ReplicationGroup updates may write a pending engine/transitEncryptionEnabled from the Modify response into spec (unverified).
  • Creating secondaries from a ReplicationGroup (globalReplicationGroupID): community#3099.

Agent-assisted: authored by an AI agent (backlog-burndown-bot) under human supervision — @gobos12

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Adds the ElastiCache Global Datastore resource, which backs cross-region
replication. The configuration is declarative apart from what the API forces
into hooks: ModifyGlobalReplicationGroup accepts only one logical change per
request, deleting a datastore requires disassociating every secondary member
first, and the Describe response carries no top-level automatic-failover
field, so that value is derived from member state.

Rebuilds the approach prototyped in aws-controllers-k8s#228
onto current main and incorporates the review findings from that PR.

Co-authored-by: NicholasBlaskey <31451191+NicholasBlaskey@users.noreply.github.com>
@ack-prow
ack-prow Bot requested review from jlbutler and sapphirew September 18, 2026 16:45
@ack-prow ack-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 18, 2026
@ack-prow

ack-prow Bot commented Sep 18, 2026

Copy link
Copy Markdown

Hi @backlog-burndown-bot. Thanks for your PR.

I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@gobos12

gobos12 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@ack-prow ack-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 29, 2026
The generated read path maps Engine, EngineVersion, CacheNodeType and
Description out of the Describe response whether or not the user declared
them. A datastore inherits the first three from its primary, so AWS reports
them for a manifest that declares none, leaving a nil-desired-versus-observed
difference no reconcile can clear: the delta never empties, every pass issues
a ModifyGlobalReplicationGroup carrying no field, and because more than one
difference is outstanding the update requeues itself indefinitely. IsSynced
reads the datastore's status, which stays steady, so the loop is silent.
Mirror each of the four only when the user manages it, as the node count and
failover fields already were.

Two e2e failures, both in the test code:

Asserting EngineVersion == "7.1" can never pass, because ElastiCache resolves
a requested version to a fully qualified one and reports that back. Compare by
major.minor, matching the rule util.EngineVersionsMatch already applies to the
delta, and give the upgrade a budget sized for a member-by-member rollout.

The suite runs under xdist LoadScheduling with more workers than tests, so
each test gets its own process and a module-scoped fixture is instantiated per
worker. Four tests requesting the primary provisioned four two-node groups
concurrently, and the contention pushed one past its budget. Fold the three
modify cases into one sequence against a single datastore, which also tests
the one-change-per-request rule against an already-modified group. Replace the
teardown's deletion waiter, which treats the pre-deletion 'available' state as
a terminal failure and leaked the group.

Co-authored-by: NicholasBlaskey <NicholasBlaskey@users.noreply.github.com>

@gobos12 gobos12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments. I also ran this PR against a real cluster (with one primary and no secondary region) to check create, update, and delete and to look for reconcile loops. I left comments in my review with the findings, but I did find one thing that was an issue with the ReplicationGroup resource as well: the RG CR breaks once it becomes a datastore member.

The ReplicationGroup resource compares its spec against AWS without regard to datastore membership. Two cases came up:

  • Parameter group (happens without any user change): joining a Global Datastore switches the member to an AWS-generated parameter group (global-datastore-<rg-id>…). On the RG's next resync, the delta shows cacheParameterGroupName: default.redis7 → global-datastore-…, so ModifyReplicationGroup is sent and AWS rejects it:

    InvalidParameterValue: Cluster [ack-grg-test-primary] is part of a global cluster [sgaui-ack-grg-test]. Request rejected.
    

    The RG CR then goes ACK.Terminal=True / Ready=False. Following the e2e test's pattern (an RG CR plus a GRG referencing it) therefore leaves the primary RG CR Terminal within one resync period.

  • Node type (after a GRG scale): after the GRG scaled members to cache.r6g.xlarge, the RG CR still declares cache.r6g.large. The rollback guard in pkg/resource/replication_group/hooks.go (Cannot update CacheNodeType, Please refer to Events…) returns a retryable error without an AWS call. The result is a retry loop that never ends, with exponential backoff; I saw it retry about once a second at first.

Aligning the RG spec with the AWS values (node type and the generated parameter group name) clears both, but users can't be expected to discover that. One suggestion: in the RG modifyDelta, when latest.ko.Status.GlobalReplicationGroupInfo is set, drop Spec.CacheNodeType and Spec.CacheParameterGroupName (and probably engine/engine version) from the delta, since the datastore owns them. The fix locus here lies in the ReplicationGroup code, so I would suggest a follow-up PR with that code fix instead of fixing it here.

Comment thread apis/v1alpha1/generator.yaml Resolved
Comment thread apis/v1alpha1/generator.yaml Resolved
Comment thread apis/v1alpha1/generator.yaml Resolved
Comment on lines +613 to +617
// setResourceFromGlobalReplicationGroup folds an Increase/Decrease/Modify
// response back into the resource. Only the identifier and status move: the
// response echoes the datastore's pre-mutation field values, and the next read is
// what establishes observed state.
func (rm *resourceManager) setResourceFromGlobalReplicationGroup(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setResourceFromGlobalReplicationGroup takes Status from the Modify response. For a description-only update, ModifyGlobalReplicationGroup responded with available, while the Describe reported primary-only. With pending == 1 there is no requeue, so the CR keeps the stale status until the 10h resync. The comment here says "the next read is what establishes observed state", but nothing schedules that read.

This can also happen when a shard change is bundled with other changes (like description), since customUpdate checks the shard count first and returns updatenodeGroupCount, leaving other changes unconsidered until the next resync.

Requeuing after any successful mutation, not only when pending > 1, would fix it. This also would've been caught if the e2e tests checked the CR's spec and status, not just the primary RG CR.

Merge upstream/main (ACK runtime and code-generator v0.66.0) and
regenerate at code-generator v0.66.0, then address the review findings.

CreateGlobalReplicationGroup reports the node type, engine and engine
version that a datastore inherits from its primary, without ever
accepting them as input. The generated create set-output copied all
three into the spec, overwriting values the user had declared and
filling in ones they left unset, which defeated the read path's
managed-only guard for anything the controller creates. Suppress the
copy with a Create-scoped set/ignore on each field, following the
Durability precedent already in this generator config.

A successful mutation now always requeues. The response echoes the
datastore's pre-mutation field values and can report a status the
subsequent Describe contradicts, so requeuing only when more of the
delta was still pending left the resource holding a stale status until
the next resync period. The shard-count path never requeued at all, and
it short-circuits the remainder of the delta.

Drop the redundant is_required tags: the generator already infers both
fields from the Create input's required list.

Assert the datastore's own spec and status in the e2e suite rather than
only the primary ReplicationGroup, which is what let both defects
above pass unnoticed.

Co-authored-by: NicholasBlaskey <31451191+NicholasBlaskey@users.noreply.github.com>
The previous regeneration rendered the service account, cluster role and
deployment with an empty service name, producing "ack--controller" where
every other elasticache artifact says "ack-elasticache-controller".
verify-code-gen caught it on five files.

build-controller-release.sh assigns SERVICE from its positional argument
without exporting it, so ack-generate -- a child process -- renders the
name from an unset environment variable. CI never hits this because it
runs `make build-controller SERVICE=elasticache`, and make exports
variables set on the command line. Regenerated the same way.

No hand-authored file changes. config/rbac/cluster-role-controller.yaml
keeps the only intended delta in these files, the new resource's own
RBAC rules.
Both delete paths checked only side effects. delete_grg waited for the AWS
datastore to disappear, and the terminal-primary case issued a delete without
checking anything at all. Neither proved the CR lost its finalizer, which is
what a converged delete looks like from the user's side.

Poll for the CR's removal in both places. The terminal case cannot reuse
delete_grg because creation never succeeded, so no datastore exists to wait on.
The comment on nextModify said engine and engine version "must travel
together", which reads as both being mandatory whenever either changes. The
rule it meant to record is narrower: the two count as one logical change rather
than two, so they may share a request under the one-change-per-request
constraint.

Nothing is mandatory when the other field is absent. A version-only bump sends
no engine, matching the request replication_group's modify path builds for the
same change, and ModifyGlobalReplicationGroupInput marks only ApplyImmediately
and GlobalReplicationGroupId required.
Polling the CR's removal already establishes that the datastore is gone. The
delete hook requeues after issuing Delete and keeps requeueing while AWS
reports deleting, so the finalizer only comes off once Describe reports
nothing. Waiting on the AWS status afterwards re-asserted a fact the first
assertion had already proven.

The global_id argument goes with it, as does the status helper's 'deleted'
branch, which no caller reaches now.
wait_replication_group_deleted hand-rolled a poll of the AWS API, and the
obvious replacement -- boto's replication_group_deleted waiter, as
test_replicationgroup.py uses it -- is no better here: its failure acceptor
treats status 'available' as fatal, so it depends on a fixed sleep being longer
than the controller's time to issue the delete.

Neither is needed. replication_group's sdkDelete requeues while the group reads
deleting, so the CR's finalizer only comes off once the group is gone from AWS,
the same invariant that makes an AWS-side check redundant in delete_grg.
Polling the CR's removal proves deletion with no race to sleep past, and uses
one mechanism for both resources.
@shabbskagalwala

Copy link
Copy Markdown

@gobos12 👋
Just wanted to check if there's any ETA on when this would be merged and available for use?

Thank you!

@gobos12

gobos12 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

@shabbskagalwala I'm working on getting a second review from another ACK maintainer. I don't have a specific ETA, but I'll stay on top of it and keep you updated on the release. Thanks for your patience!

@knottnt knottnt self-assigned this Oct 20, 2026

@knottnt knottnt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gobos12 for this resource is there a plan to add support for setting a spec.globalResourceGroupID field to the ReplicationGroupCRD modeled from the CreateResourceGroup API? Without that follow-up it seems that users still wouldn't be able to assign read replicas via ACK.

Comment thread generator.yaml Outdated
Comment on lines +592 to +593
tags:
ignore: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing the AddTagsToResource API it seems that GlobalReplciationGroups can be tagged. Any reason we're using tags:ignore here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CreateGlobalReplicationGroup accepts no tags at the pinned elasticache v1.54.0, so there's no Tags field for the generator to manage and ignore: true is what makes generation succeed. Tags were added to that input by the 2026-09-24 client regen, so this is a pin limitation — once the SDK is bumped, removing the ignore is a one-line change, which I'd keep to its own PR. Updated the comment in f197ac3 to say that.

Comment thread generator.yaml Outdated
Comment on lines +560 to +563
- ReplicationGroupNotFoundFault
# A parameter group named for a major engine upgrade that does not exist is
# a spec error, not something retrying fixes. The code has no Fault suffix.
- CacheParameterGroupNotFound

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there scenarios where these could be transient states? For example, if the primary replication group's CR is applied at the same time the GlobalReplicationGroup could race it and see the ReplicationGroupNotFoundFault error. For CacheParameterGroupNotFound, Elasticache does support user created CacheParameterGroups. If those can be referenced by the global replication group then the same could apply to CacheParameterGroupNotFound.

ReplicationGroupNotFoundFault and CacheParameterGroupNotFound name a
dependency rather than the datastore itself, so either can mean "not
created yet" rather than "wrong". Treating them as terminal stranded the
resource until its spec changed, with no recovery for a primary or
parameter group created concurrently; a reference orders the primary,
but a literal ID has no such ordering. They now retry, and
InvalidParameterValue still catches a genuinely bad value.

Also records why tags are ignored: CreateGlobalReplicationGroup accepts
no tags at the pinned SDK version, so no Tags field exists for the
generator to manage.
@gobos12

gobos12 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

/test all

@ack-prow

ack-prow Bot commented Oct 21, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: backlog-burndown-bot
Once this PR has been reviewed and has the lgtm label, please ask for approval from knottnt. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@backlog-burndown-bot

Copy link
Copy Markdown
Contributor Author

/test all

ReplicationGroupNotFoundFault is no longer a terminal code, so a datastore
naming a nonexistent primary now surfaces ACK.Recoverable rather than
ACK.Terminal. The primary's ID is user-chosen, so a datastore and its primary
can legitimately be applied together and the error can mean "not created yet"
as readily as "wrong" -- stranding the resource until its spec changes is the
wrong response.

Asserts the condition message carries the error code so the test cannot pass on
an unrelated error, and keeps the check that such a resource is still deletable.
@backlog-burndown-bot

Copy link
Copy Markdown
Contributor Author

/retest

Once a ReplicationGroup is a Global Datastore member, ModifyReplicationGroup
rejects cacheNodeType, cacheParameterGroupName, engineVersion,
automaticFailoverEnabled and autoMinorVersionUpgrade with
InvalidParameterValue, even unchanged. Creating a GlobalReplicationGroup
swaps the primary's parameter group to global-datastore-*, so the primary's
next reconcile tried to revert it and went Terminal, which then blocked the
GlobalReplicationGroup through primaryReplicationGroupRef.

Keyed off status.globalReplicationGroupInfo, not role, since roles swap on
failover and a primary joins without any change to its own spec:
- drop those fields from the delta, and adopt the observed node type and
  parameter group into spec rather than leaving stale values to be
  re-applied after the group leaves the datastore
- strip them from the generated ModifyReplicationGroup input, which sends
  every spec field regardless of the delta
@backlog-burndown-bot

Copy link
Copy Markdown
Contributor Author

/retest

ack-prow Bot pushed a commit that referenced this pull request Oct 26, 2026
Issue #, if available: prerequisite for #237

Description of changes:

Bumps the core `aws-sdk-go-v2` from v1.41.10 to v1.47.4 (and `service/elasticache` v1.54.0 -> v1.64.1), then regenerates.

The reason for the bump is that `CreateGlobalReplicationGroupInput` only gained `Tags` in elasticache v1.62.0. #237 currently sets `tags: ignore: true` on `GlobalReplicationGroup` for that reason, and will enable tags once this merges.

This PR adds no new API surface. The fields the newer model adds to ServerlessCache (`connectionType` and `status.migration`) are in `ignore.field_paths`, and support for them can follow in a separate PR. The only CRD changes are doc text, such as the `Tag` description no longer excluding global replication groups. The new `PrepareMigration` / `StopMigration` operations aren't CRUD verbs, so no new CRD is generated. A few unused types from those operations do appear in `types.go`/`enums.go`, but no CRD references them.

Checks: `crd-compat-check` reports non-breaking changes only across all 20 CRDs. Build and unit tests pass. The regen used Go 1.27.1 (the CI pin) and code-generator v0.66.0, matching the base's `build_hash`. `ATTRIBUTION.md` is unchanged, since it lists modules, not versions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
aws-sdk-go-v2 elasticache v1.62.0 models Tags on CreateGlobalReplicationGroup.
Drop tags: ignore, read tags back with ListTagsForResource, and sync changes
through AddTagsToResource/RemoveTagsFromResource since
ModifyGlobalReplicationGroup does not accept them.
@gobos12

gobos12 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Live test: #237 @ 1bdd88e with feat/3099-rg-global-replication-group-id @ 35e7a62 on top

Setup:

  • Primary RG in us-west-2 (redis 7.0, cache.r6g.large, cluster mode disabled, 1 replica).
  • Secondaries in us-east-1 (via globalReplicationGroupRef) and us-east-2 (via explicit globalReplicationGroupID).
  • All three driven by one local controller.
  • go build and unit tests pass. Regenerating with code-generator v0.67.0 shows no drift.

39187ce fixes the member Terminal problem from the earlier round. These all worked with no Terminal condition on any member:

  • All three members join and stay Synced. The global-datastore-* parameter group swap is adopted into spec, and no ModifyReplicationGroup calls go out.
  • Description edits apply on both the primary and a secondary. Tags and a replica-count increase also apply.
  • GRG scale cache.r6g.large → cache.r6g.xlarge: members picked up the new cacheNodeType within about 15 min, with no restart.
  • GRG engine upgrade 7.0 → 7.1.
  • Out-of-band failover to us-east-1: member roles in status.globalReplicationGroupInfo refreshed within about 7 min.
  • GRG delete after failover: it disassociated the remaining secondary and deleted the datastore. The new primary was kept.

Issues

  1. engineVersion isn't adopted, so a member goes Terminal after it leaves the datastore. reconcileDatastoreOwnedFields drops the Spec.EngineVersion delta but only copies the observed cacheNodeType and cacheParameterGroupName into spec. So a node-type edit on a member is reverted to the observed value, but an engineVersion edit (or a GRG-driven upgrade) leaves spec stale.
    • After the GRG upgrade 7.0 → 7.1, the primary's spec still said "7.0" while AWS ran 7.1.0, and the CR stayed Synced.
    • When the GRG was deleted and the RG became standalone, the controller sent EngineVersion=7.0 and got InvalidParameterCombination: Cannot modify redis from 7.1.0 to redis version 7.0.7, so the CR went Terminal.
    • A stale value higher than the observed one would instead trigger an upgrade nobody asked for.
    • Suggested fix: when engineVersion is declared, adopt the observed value, as is already done for cacheNodeType. Do the same for automaticFailoverEnabled. Feat/autominorupgrade #245 avoids the downgrade for same-major minor upgrades, but not for major-version or engine changes.
  2. status.members on the GRG CR doesn't refresh. After the failover, status.members still showed the old roles 30 min later, because GlobalReplicationGroup has no requeue_on_success_seconds. spec.primaryReplicationGroupRef also keeps naming the old primary, with no condition saying so.
  3. Worth documenting: the primary's state blocks the GRG. A non-Synced primary RG CR blocks the GRG through primaryReplicationGroupRef, even when it's transient (for us, an ExpiredToken). That in turn blocks any secondary that uses globalReplicationGroupRef. This is standard ACK reference behaviour, but the chain isn't obvious.

For the stacked secondaries branch (feat/3099)

Disassociate-before-delete works:

  • Deleting a secondary's CR disassociates it through its own region's client, then deletes it.
  • Deleting the CR of the current primary requeues with "delete the GlobalReplicationGroup first" and completes once the GRG is gone.

Other notes:

  • Re-create would be blocked (not run live, inferred from the code). After create, the controller writes engine and cacheNodeType from the Create response into the secondary's spec. Those are the fields rejectInheritedFieldsOnDatastoreJoin rejects. Re-creating from that spec (after an out-of-band delete, or restoring the CR from a backup) would hit the same Terminal error as a bad manifest.
  • A typo'd globalReplicationGroupID can't be fixed in place, even though the RG was never created (self == oldSelf). The CR has to be deleted and recreated. Probably fine, but worth mentioning in the docs.
  • Message nits:
    • The replica-count guard error starts with spec.globalReplicationGroupID:.
    • The Synced message on a primary that's waiting for the GRG to be deleted says "currently being disassociated", which isn't what's happening.

A Global Datastore member now copies the observed engineVersion and
automatic failover setting into spec, as it already did for node type and
parameter group, so a member that leaves the datastore no longer sends a
stale engine version (downgrade rejected, CR Terminal) or failover setting.

GlobalReplicationGroup requeues every 60s on success so status.members
reflects out-of-band failovers instead of waiting for the default resync.
requeue_on_success_seconds makes the generated updateConditions add
ResourceSynced=True after any successful call, so the runtime skips
IsSynced and the synced.when rule never runs: a datastore still
"creating" reported Synced, with no lastTransitionTime. Re-apply the
rule in a CustomUpdateConditions hook, as sagemaker-controller does for
its requeue-on-success resources.
Comment thread pkg/resource/replication_group/hooks.go Resolved
A Global Datastore member now adopts the observed engine like the other
datastore-owned fields, and strips it from ModifyReplicationGroup. While
the group is a member it carries an ACK.Advisory condition naming the
datastore and the fields it manages, and the update path logs which
datastore-owned differences it adopted.
Comment on lines +106 to +109
// Describe response cannot map directly. Both are populated only when the user
// actually manages the field: writing an observed value into a Spec field left
// unset would make the controller take ownership of it and then patch it back
// into the user's manifest.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are populated only when the user actually manages the field: writing an observed value into a Spec field left unset would make the controller take ownership of it and then patch it back into the user's manifest.

Pretty sure, this kind of filtering would also prevent these values from being imported during adoption. Also, during non-adoption flows we don't write back values directly from sdkFind when late-initialization isn't being used.


// newCreateRequestPayload returns an SDK-specific struct for the HTTP request
// payload of the Create API call for the resource
func (rm *resourceManager) newCreateRequestPayload(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that a number of spec fields aren't settable in the Create payload. If one of them is set at create time do we need to force the resource to Synced:False to force a requeue and update operation similar to what we do in the S3 bucket?

https://github.com/aws-controllers-k8s/s3-controller/blob/main/templates/hooks/bucket/sdk_create_post_set_output.go.tpl

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed: Create reports creating, so CustomUpdateConditions keeps it Synced=False (and the 60s requeue-on-success brings it back regardless). The create-unsupported fields aren't copied back from the Create response, so declared values stay in desired and any difference from the inherited value goes through the normal update once the datastore settles. The exception is cacheParameterGroupName, which is only sent alongside an engine upgrade.

ko.Status.Status = nil
}
if resp.GlobalReplicationGroup.TransitEncryptionEnabled != nil {
ko.Status.TransitEncryptionEnabled = resp.GlobalReplicationGroup.TransitEncryptionEnabled

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: Just a sanity check. Presumably, these status values are coming from the primary replication group specified by Spec.PrimaryReplicationGroupId. After the GlobalReplicationGroup has been created are these boolean fields still controlled by the primary or made immutable?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're status-only and re-read from Describe each reconcile. Tested on a dev cluster: AWS rejects any TLS change on a member, so transitEncryptionEnabled is now datastore-owned on the ReplicationGroup side (5bf3425). Auth tokens can still be set or rotated on a TLS member, but DescribeGlobalReplicationGroups keeps reporting AuthTokenEnabled: false afterwards, so the GRG status reflects that; noted under known limitations.

Stop filtering read-path Spec values by whether the manifest declares
them. The runtime writes sdkFind's Spec into the CR only on adoption or
late-init, so the filter guarded nothing in normal reconciles and kept
adoption from importing the datastore's configuration. modifyDelta now
drops a difference on an undeclared inherited field instead, and an
update with nothing modifiable returns desired rather than latest.
AWS rejects any in-transit encryption change on a Global Datastore member,
so drop TransitEncryptionEnabled from a member's delta, adopt the observed
value into spec, and strip TransitEncryptionEnabled/TransitEncryptionMode
from the ModifyReplicationGroup request.
…owned

AWS rejects resharding and durability changes on a Global Datastore member,
and primaryClusterID can only change with automatic failover off, which a
member cannot turn off. Adopt the observed shard count (refreshing the
last-requested annotations so the resharding-rollback guard does not go
Terminal) and durability, drop primaryClusterID from the delta, and strip
Durability and PrimaryClusterId from the ModifyReplicationGroup request.
@backlog-burndown-bot

Copy link
Copy Markdown
Contributor Author

/retest

Comment thread apis/v1alpha1/generator.yaml Resolved
Comment on lines +501 to +505
PrimaryReplicationGroupId:
is_immutable: true
references:
resource: ReplicationGroup
path: Spec.ReplicationGroupID

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was testing failovers on one of my clusters and found that this ref still names the original primary after a failover occurs. Since the field is immutable, it can't be corrected. Nothing flag that it no longer matches, and the only way to know which member is the primary is by checking status. It becomes a bigger problem if the ex-primary-now-secondary gets disassociated and deleted; the references can't resolve and this CR ends up stuck in ACK.ReferencesResolved=Unknown. Related: because the ref is resolved on every reconcile, it also fails whenever the primary RG CR isn't Synced, which includes during the GRG's own scale or upgrade putting the primary RG into modifying, so a ref-based GRG stops observing its own changes until the primary re-syncs.

Updating the spec to match the primary member in status would cause more problems than it would solves (the immutability rule rejects it, GitOps would drift after every failover, and the new primary may not be an ACK-managed RG). I think the best course of action would be to treat this as a create-time input only, and stop resolving it afterwards. We should also show the current datastore members from status.members in a printer column, to make the current primary/secondaries easy to see.

Comment on lines +501 to +505
PrimaryReplicationGroupId:
is_immutable: true
references:
resource: ReplicationGroup
path: Spec.ReplicationGroupID

@gobos12 gobos12 Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another issue with this field: when a GRG is adopted without the primaryReplicationGroupID/Ref in spec, the field is never imported. Nothing ever fills this field on read, as DescribeGlobalReplicationGroups has no primary-ID field, so the spec that adoption imports never gets it. This could be fixed by filling it on read from the primary member via a custom describe hook, but that would record the current primary rather than the one the datastore was created from. Another solution here is the same as my other comment: treat this field as create-time only. That way, both the ID-or-ref validation and the ref resolution only apply before the GRG exists. But that also means the field wouldn't get written into spec on adoption, and that's not ideal.

Comment on lines +38 to +47
var datastoreOwnedSpecPaths = []string{
"Spec.CacheNodeType",
"Spec.CacheParameterGroupName",
"Spec.Engine",
"Spec.EngineVersion",
"Spec.AutomaticFailoverEnabled",
"Spec.TransitEncryptionEnabled",
"Spec.Durability",
"Spec.NumNodeGroups",
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm realizing that these would cause GitOps drift if we write datastore-owned fields into spec. We've seen this kind of pattern before in this controller for other resources, so I think we should take it into consideration for this PR, as that back-and-forth with this many fields can cause a lot of churn. I wonder if there's a way to treat datastore members as compare: is_ignore, but only while they are members. We'd have to do an adoption if they're ever disassociated from the datastore though, since the spec would go stale pretty quickly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could also have a documentation.yaml line to specify that datastore-owned fields in spec won't match status if they are updated through the datastore. Also look into adding an annotation for replication groups owned by datastores so we can ignore drift on those resources even after they've been disassociated with the datastore.

This is an alternative to writing back to spec that's causing the flip-flopping with GitOps tools. Also an alternative to the solution I mention in this comment, since that one-time adoption just get written back by GitOps.

…y at create

ReplicationGroup: while the group is a Global Datastore member, drop the
datastore-owned differences in modifyDelta instead of copying AWS values
into spec, restore the declared values on update results, and skip the
shard and failover paths that compare desired against latest directly.

GlobalReplicationGroup: stop resolving primaryReplicationGroupRef once the
datastore exists, fill primaryReplicationGroupID on adoption from the
PRIMARY member, and add a PRIMARY printer column.

Document both in a new documentation.yaml.
…-global-replication-group

aws-controllers-k8s#245 makes autoMinorVersionUpgrade a ReplicationGroup spec field. AWS rejects
it on a Global Datastore member, so it joins the datastore-owned set.

@gobos12 gobos12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the GRG is created (or fails over, or is deleted), nothing reconciles the member RG CRs, so their status.globalReplicationGroupInfo, role and the GlobalDatastoreMember advisory stay stale until the next resync, which is 10h by default. Datastore-owned fields are still protected, because any spec edit re-reads membership from AWS. But the CR doesn't show it's a member, and a FieldExport of the role is wrong after a failover. Can we set requeue_on_success_seconds on ReplicationGroup so membership changes show up within a bounded interval? Something like 600s seems fine: it applies to every RG, not just members, but that's one extra read cycle per RG every 10 minutes, and operators can still override it per kind with --reconcile-resource-resync-seconds.

Comment thread pkg/resource/replication_group/hooks.go Resolved
…y on updates

Joining, leaving or failing over a Global Datastore happens outside the ReplicationGroup, so re-read it every 600s to keep globalReplicationGroupInfo and the member advisory current. Update results start from desired, whose conditions the runtime cleared, so set the advisory there too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Indicates a non-member PR verified by an org member that is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support GlobalReplicationGroup in elasticache-controller

4 participants