Skip to content

[elasticache] ReplicationGroup stops reconciling once it joins a Global Datastore #3091

Open

Description

@gobos12

Describe the bug

A ReplicationGroup managed by ACK breaks as soon as it becomes a member of an ElastiCache Global Datastore. The resource goes ACK.Terminal=True / Ready=False and stops converging, without the user changing anything in the manifest.

The ReplicationGroup resource compares its spec against the AWS state with no regard for datastore membership, but a Global Datastore owns several of those fields and rejects member-level modification of them. Two cases, both observed on a live cluster:

1. Cache parameter group — happens with no user action at all.

Joining a Global Datastore switches the member to an AWS-generated parameter group (global-datastore-…). On the next resync the delta shows cacheParameterGroupName: default.redis7 → global-datastore-…, a ModifyReplicationGroup is issued, and AWS rejects it:

InvalidParameterValue: Cluster [my-primary] is part of a global cluster [my-datastore]. Request rejected.

This needs nothing from the user because the field is late-initialized: the pre-join value is written into spec automatically, so the spec is guaranteed to disagree with AWS the moment the datastore takes the field over.

2. Node type — after a datastore-level scale.

Scaling the datastore moves members to a new node type (e.g. cache.r6g.large → cache.r6g.xlarge) while the RG's spec still declares the old one. The scale-down rollback guard then rejects the difference locally, with no AWS call:

InvalidParameterCombination: Cannot update CacheNodeType, Please refer to Events for more details

Aligning the spec by hand with the AWS values clears both, but users have no way to discover that is what is required.

Root cause

Parameter group: setCacheParameterGroup copies the observed parameter group into latest.Spec on every read, and the field carries late_initialize (added for #3000) so the observed value is also persisted into desired.Spec. Once the datastore renames it, the two disagree permanently. modifyDelta does not strip the difference, the modify gate fires on Spec.CacheParameterGroupName, and the payload sends the field because desired is now non-nil.

Node type: the rollback guard treats "spec differs from AWS, but we did not request this" as a failed scale. cacheNodeTypeRequiresUpdate compares desired against the last-requested annotation, which still holds the original value because the RG never requested the change — the datastore made it. InvalidParameterCombination is in this resource's terminal_codes, so this path is terminal as well, despite making no API call.

Steps to reproduce

No unreleased resource is needed — the member RG can be ACK-managed while the datastore is created elsewhere:

  1. Create (or adopt) a ReplicationGroup with ACK. Any manifest that sets securityGroupIDs or engineVersion reaches the affected code path; a manifest setting neither short-circuits before the ModifyReplicationGroup call.
  2. Create a Global Datastore with that replication group as its primary, from the console or aws elasticache create-global-replication-group.
  3. Watch the RG CR over the next reconcile.

Observed: ACK.Terminal=True with the InvalidParameterValue message above, and the parameter-group delta repeating on every reconcile.

Expected outcome

A ReplicationGroup that is a Global Datastore member stays Synced and does not go Terminal, because the fields the datastore owns are not the RG's to reconcile.

Suggested fix

status.globalReplicationGroupInfo is already populated on the RG from both the Describe and Modify read paths, so membership can be detected with no new API surface. In modifyDelta, when latest.ko.Status.GlobalReplicationGroupInfo is non-nil, drop the datastore-owned fields from the delta — Spec.CacheNodeType and Spec.CacheParameterGroupName at minimum. This is the shape modifyDelta already uses for Spec.PreferredMaintenanceWindow.

Two points for maintainer input:

  • engine / engineVersion. A datastore-level engine upgrade propagates to members, which should produce the same dead-end delta. Read from the code rather than observed, and these two are entangled with the RG's existing engine-upgrade path, so suppressing them may want a narrower condition than the other fields.
  • Visibility. Silently dropping the difference leaves the spec disagreeing with reality with nothing to explain why. An event, or documenting the fields as datastore-owned, may be worth pairing with the suppression.

Environment

  • Kubernetes version: 1.35 (EKS)
  • Using EKS: yes, 1.35
  • AWS service targeted: ElastiCache (ReplicationGroup)

Activity

  1. self-assigned this
    on Oct 15, 2026
  2. added
    kind/bugCategorizes issue or PR as related to a bug.
    service/elasticacheIndicates issues or PRs that are related to elasticache-controller.
    on Oct 15, 2026
  3. github-actions commented on Oct 15, 2026

    @github-actions

    Hello @gobos12 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

    We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

    You can find more information about ACK on our website.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.service/elasticacheIndicates issues or PRs that are related to elasticache-controller.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions