Repository navigation
Add GlobalReplicationGroup resource support - #237
backlog-burndown-bot wants to merge 24 commits into
Conversation
Adds the ElastiCache Global Datastore resource, which backs cross-region replication. The configuration is declarative apart from what the API forces into hooks: ModifyGlobalReplicationGroup accepts only one logical change per request, deleting a datastore requires disassociating every secondary member first, and the Describe response carries no top-level automatic-failover field, so that value is derived from member state. Rebuilds the approach prototyped in aws-controllers-k8s#228 onto current main and incorporates the review findings from that PR. Co-authored-by: NicholasBlaskey <31451191+NicholasBlaskey@users.noreply.github.com>
|
Hi @backlog-burndown-bot. Thanks for your PR. I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
The generated read path maps Engine, EngineVersion, CacheNodeType and Description out of the Describe response whether or not the user declared them. A datastore inherits the first three from its primary, so AWS reports them for a manifest that declares none, leaving a nil-desired-versus-observed difference no reconcile can clear: the delta never empties, every pass issues a ModifyGlobalReplicationGroup carrying no field, and because more than one difference is outstanding the update requeues itself indefinitely. IsSynced reads the datastore's status, which stays steady, so the loop is silent. Mirror each of the four only when the user manages it, as the node count and failover fields already were. Two e2e failures, both in the test code: Asserting EngineVersion == "7.1" can never pass, because ElastiCache resolves a requested version to a fully qualified one and reports that back. Compare by major.minor, matching the rule util.EngineVersionsMatch already applies to the delta, and give the upgrade a budget sized for a member-by-member rollout. The suite runs under xdist LoadScheduling with more workers than tests, so each test gets its own process and a module-scoped fixture is instantiated per worker. Four tests requesting the primary provisioned four two-node groups concurrently, and the contention pushed one past its budget. Fold the three modify cases into one sequence against a single datastore, which also tests the one-change-per-request rule against an already-modified group. Replace the teardown's deletion waiter, which treats the pre-deletion 'available' state as a terminal failure and leaked the group. Co-authored-by: NicholasBlaskey <NicholasBlaskey@users.noreply.github.com>
gobos12
left a comment
There was a problem hiding this comment.
Left a few comments. I also ran this PR against a real cluster (with one primary and no secondary region) to check create, update, and delete and to look for reconcile loops. I left comments in my review with the findings, but I did find one thing that was an issue with the ReplicationGroup resource as well: the RG CR breaks once it becomes a datastore member.
The ReplicationGroup resource compares its spec against AWS without regard to datastore membership. Two cases came up:
-
Parameter group (happens without any user change): joining a Global Datastore switches the member to an AWS-generated parameter group (
global-datastore-<rg-id>…). On the RG's next resync, the delta showscacheParameterGroupName: default.redis7 → global-datastore-…, soModifyReplicationGroupis sent and AWS rejects it:InvalidParameterValue: Cluster [ack-grg-test-primary] is part of a global cluster [sgaui-ack-grg-test]. Request rejected.The RG CR then goes
ACK.Terminal=True/Ready=False. Following the e2e test's pattern (an RG CR plus a GRG referencing it) therefore leaves the primary RG CR Terminal within one resync period. -
Node type (after a GRG scale): after the GRG scaled members to
cache.r6g.xlarge, the RG CR still declarescache.r6g.large. The rollback guard inpkg/resource/replication_group/hooks.go(Cannot update CacheNodeType, Please refer to Events…) returns a retryable error without an AWS call. The result is a retry loop that never ends, with exponential backoff; I saw it retry about once a second at first.
Aligning the RG spec with the AWS values (node type and the generated parameter group name) clears both, but users can't be expected to discover that. One suggestion: in the RG modifyDelta, when latest.ko.Status.GlobalReplicationGroupInfo is set, drop Spec.CacheNodeType and Spec.CacheParameterGroupName (and probably engine/engine version) from the delta, since the datastore owns them. The fix locus here lies in the ReplicationGroup code, so I would suggest a follow-up PR with that code fix instead of fixing it here.
| // setResourceFromGlobalReplicationGroup folds an Increase/Decrease/Modify | ||
| // response back into the resource. Only the identifier and status move: the | ||
| // response echoes the datastore's pre-mutation field values, and the next read is | ||
| // what establishes observed state. | ||
| func (rm *resourceManager) setResourceFromGlobalReplicationGroup( |
There was a problem hiding this comment.
setResourceFromGlobalReplicationGroup takes Status from the Modify response. For a description-only update, ModifyGlobalReplicationGroup responded with available, while the Describe reported primary-only. With pending == 1 there is no requeue, so the CR keeps the stale status until the 10h resync. The comment here says "the next read is what establishes observed state", but nothing schedules that read.
This can also happen when a shard change is bundled with other changes (like description), since customUpdate checks the shard count first and returns updatenodeGroupCount, leaving other changes unconsidered until the next resync.
Requeuing after any successful mutation, not only when pending > 1, would fix it. This also would've been caught if the e2e tests checked the CR's spec and status, not just the primary RG CR.
Merge upstream/main (ACK runtime and code-generator v0.66.0) and regenerate at code-generator v0.66.0, then address the review findings. CreateGlobalReplicationGroup reports the node type, engine and engine version that a datastore inherits from its primary, without ever accepting them as input. The generated create set-output copied all three into the spec, overwriting values the user had declared and filling in ones they left unset, which defeated the read path's managed-only guard for anything the controller creates. Suppress the copy with a Create-scoped set/ignore on each field, following the Durability precedent already in this generator config. A successful mutation now always requeues. The response echoes the datastore's pre-mutation field values and can report a status the subsequent Describe contradicts, so requeuing only when more of the delta was still pending left the resource holding a stale status until the next resync period. The shard-count path never requeued at all, and it short-circuits the remainder of the delta. Drop the redundant is_required tags: the generator already infers both fields from the Create input's required list. Assert the datastore's own spec and status in the e2e suite rather than only the primary ReplicationGroup, which is what let both defects above pass unnoticed. Co-authored-by: NicholasBlaskey <31451191+NicholasBlaskey@users.noreply.github.com>
The previous regeneration rendered the service account, cluster role and deployment with an empty service name, producing "ack--controller" where every other elasticache artifact says "ack-elasticache-controller". verify-code-gen caught it on five files. build-controller-release.sh assigns SERVICE from its positional argument without exporting it, so ack-generate -- a child process -- renders the name from an unset environment variable. CI never hits this because it runs `make build-controller SERVICE=elasticache`, and make exports variables set on the command line. Regenerated the same way. No hand-authored file changes. config/rbac/cluster-role-controller.yaml keeps the only intended delta in these files, the new resource's own RBAC rules.
Both delete paths checked only side effects. delete_grg waited for the AWS datastore to disappear, and the terminal-primary case issued a delete without checking anything at all. Neither proved the CR lost its finalizer, which is what a converged delete looks like from the user's side. Poll for the CR's removal in both places. The terminal case cannot reuse delete_grg because creation never succeeded, so no datastore exists to wait on.
The comment on nextModify said engine and engine version "must travel together", which reads as both being mandatory whenever either changes. The rule it meant to record is narrower: the two count as one logical change rather than two, so they may share a request under the one-change-per-request constraint. Nothing is mandatory when the other field is absent. A version-only bump sends no engine, matching the request replication_group's modify path builds for the same change, and ModifyGlobalReplicationGroupInput marks only ApplyImmediately and GlobalReplicationGroupId required.
Polling the CR's removal already establishes that the datastore is gone. The delete hook requeues after issuing Delete and keeps requeueing while AWS reports deleting, so the finalizer only comes off once Describe reports nothing. Waiting on the AWS status afterwards re-asserted a fact the first assertion had already proven. The global_id argument goes with it, as does the status helper's 'deleted' branch, which no caller reaches now.
wait_replication_group_deleted hand-rolled a poll of the AWS API, and the obvious replacement -- boto's replication_group_deleted waiter, as test_replicationgroup.py uses it -- is no better here: its failure acceptor treats status 'available' as fatal, so it depends on a fixed sleep being longer than the controller's time to issue the delete. Neither is needed. replication_group's sdkDelete requeues while the group reads deleting, so the CR's finalizer only comes off once the group is gone from AWS, the same invariant that makes an AWS-side check redundant in delete_grg. Polling the CR's removal proves deletion with no race to sleep past, and uses one mechanism for both resources.
|
@gobos12 👋 Thank you! |
|
@shabbskagalwala I'm working on getting a second review from another ACK maintainer. I don't have a specific ETA, but I'll stay on top of it and keep you updated on the release. Thanks for your patience! |
knottnt
left a comment
There was a problem hiding this comment.
@gobos12 for this resource is there a plan to add support for setting a spec.globalResourceGroupID field to the ReplicationGroupCRD modeled from the CreateResourceGroup API? Without that follow-up it seems that users still wouldn't be able to assign read replicas via ACK.
| tags: | ||
| ignore: true |
There was a problem hiding this comment.
Testing the AddTagsToResource API it seems that GlobalReplciationGroups can be tagged. Any reason we're using tags:ignore here?
There was a problem hiding this comment.
CreateGlobalReplicationGroup accepts no tags at the pinned elasticache v1.54.0, so there's no Tags field for the generator to manage and ignore: true is what makes generation succeed. Tags were added to that input by the 2026-09-24 client regen, so this is a pin limitation — once the SDK is bumped, removing the ignore is a one-line change, which I'd keep to its own PR. Updated the comment in f197ac3 to say that.
| - ReplicationGroupNotFoundFault | ||
| # A parameter group named for a major engine upgrade that does not exist is | ||
| # a spec error, not something retrying fixes. The code has no Fault suffix. | ||
| - CacheParameterGroupNotFound |
There was a problem hiding this comment.
Are there scenarios where these could be transient states? For example, if the primary replication group's CR is applied at the same time the GlobalReplicationGroup could race it and see the ReplicationGroupNotFoundFault error. For CacheParameterGroupNotFound, Elasticache does support user created CacheParameterGroups. If those can be referenced by the global replication group then the same could apply to CacheParameterGroupNotFound.
ReplicationGroupNotFoundFault and CacheParameterGroupNotFound name a dependency rather than the datastore itself, so either can mean "not created yet" rather than "wrong". Treating them as terminal stranded the resource until its spec changed, with no recovery for a primary or parameter group created concurrently; a reference orders the primary, but a literal ID has no such ordering. They now retry, and InvalidParameterValue still catches a genuinely bad value. Also records why tags are ignored: CreateGlobalReplicationGroup accepts no tags at the pinned SDK version, so no Tags field exists for the generator to manage.
|
/test all |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: backlog-burndown-bot The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/test all |
ReplicationGroupNotFoundFault is no longer a terminal code, so a datastore naming a nonexistent primary now surfaces ACK.Recoverable rather than ACK.Terminal. The primary's ID is user-chosen, so a datastore and its primary can legitimately be applied together and the error can mean "not created yet" as readily as "wrong" -- stranding the resource until its spec changes is the wrong response. Asserts the condition message carries the error code so the test cannot pass on an unrelated error, and keeps the check that such a resource is still deletable.
|
/retest |
Once a ReplicationGroup is a Global Datastore member, ModifyReplicationGroup rejects cacheNodeType, cacheParameterGroupName, engineVersion, automaticFailoverEnabled and autoMinorVersionUpgrade with InvalidParameterValue, even unchanged. Creating a GlobalReplicationGroup swaps the primary's parameter group to global-datastore-*, so the primary's next reconcile tried to revert it and went Terminal, which then blocked the GlobalReplicationGroup through primaryReplicationGroupRef. Keyed off status.globalReplicationGroupInfo, not role, since roles swap on failover and a primary joins without any change to its own spec: - drop those fields from the delta, and adopt the observed node type and parameter group into spec rather than leaving stale values to be re-applied after the group leaves the datastore - strip them from the generated ModifyReplicationGroup input, which sends every spec field regardless of the delta
|
/retest |
Issue #, if available: prerequisite for #237 Description of changes: Bumps the core `aws-sdk-go-v2` from v1.41.10 to v1.47.4 (and `service/elasticache` v1.54.0 -> v1.64.1), then regenerates. The reason for the bump is that `CreateGlobalReplicationGroupInput` only gained `Tags` in elasticache v1.62.0. #237 currently sets `tags: ignore: true` on `GlobalReplicationGroup` for that reason, and will enable tags once this merges. This PR adds no new API surface. The fields the newer model adds to ServerlessCache (`connectionType` and `status.migration`) are in `ignore.field_paths`, and support for them can follow in a separate PR. The only CRD changes are doc text, such as the `Tag` description no longer excluding global replication groups. The new `PrepareMigration` / `StopMigration` operations aren't CRUD verbs, so no new CRD is generated. A few unused types from those operations do appear in `types.go`/`enums.go`, but no CRD references them. Checks: `crd-compat-check` reports non-breaking changes only across all 20 CRDs. Build and unit tests pass. The regen used Go 1.27.1 (the CI pin) and code-generator v0.66.0, matching the base's `build_hash`. `ATTRIBUTION.md` is unchanged, since it lists modules, not versions. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
…into feat/1048-global-replication-group
aws-sdk-go-v2 elasticache v1.62.0 models Tags on CreateGlobalReplicationGroup. Drop tags: ignore, read tags back with ListTagsForResource, and sync changes through AddTagsToResource/RemoveTagsFromResource since ModifyGlobalReplicationGroup does not accept them.
|
Live test: #237 @ Setup:
Issues
For the stacked secondaries branch (
|
A Global Datastore member now copies the observed engineVersion and automatic failover setting into spec, as it already did for node type and parameter group, so a member that leaves the datastore no longer sends a stale engine version (downgrade rejected, CR Terminal) or failover setting. GlobalReplicationGroup requeues every 60s on success so status.members reflects out-of-band failovers instead of waiting for the default resync.
requeue_on_success_seconds makes the generated updateConditions add ResourceSynced=True after any successful call, so the runtime skips IsSynced and the synced.when rule never runs: a datastore still "creating" reported Synced, with no lastTransitionTime. Re-apply the rule in a CustomUpdateConditions hook, as sagemaker-controller does for its requeue-on-success resources.
A Global Datastore member now adopts the observed engine like the other datastore-owned fields, and strips it from ModifyReplicationGroup. While the group is a member it carries an ACK.Advisory condition naming the datastore and the fields it manages, and the update path logs which datastore-owned differences it adopted.
| // Describe response cannot map directly. Both are populated only when the user | ||
| // actually manages the field: writing an observed value into a Spec field left | ||
| // unset would make the controller take ownership of it and then patch it back | ||
| // into the user's manifest. |
There was a problem hiding this comment.
Both are populated only when the user actually manages the field: writing an observed value into a Spec field left unset would make the controller take ownership of it and then patch it back into the user's manifest.
Pretty sure, this kind of filtering would also prevent these values from being imported during adoption. Also, during non-adoption flows we don't write back values directly from sdkFind when late-initialization isn't being used.
|
|
||
| // newCreateRequestPayload returns an SDK-specific struct for the HTTP request | ||
| // payload of the Create API call for the resource | ||
| func (rm *resourceManager) newCreateRequestPayload( |
There was a problem hiding this comment.
I see that a number of spec fields aren't settable in the Create payload. If one of them is set at create time do we need to force the resource to Synced:False to force a requeue and update operation similar to what we do in the S3 bucket?
There was a problem hiding this comment.
Not needed: Create reports creating, so CustomUpdateConditions keeps it Synced=False (and the 60s requeue-on-success brings it back regardless). The create-unsupported fields aren't copied back from the Create response, so declared values stay in desired and any difference from the inherited value goes through the normal update once the datastore settles. The exception is cacheParameterGroupName, which is only sent alongside an engine upgrade.
| ko.Status.Status = nil | ||
| } | ||
| if resp.GlobalReplicationGroup.TransitEncryptionEnabled != nil { | ||
| ko.Status.TransitEncryptionEnabled = resp.GlobalReplicationGroup.TransitEncryptionEnabled |
There was a problem hiding this comment.
Q: Just a sanity check. Presumably, these status values are coming from the primary replication group specified by Spec.PrimaryReplicationGroupId. After the GlobalReplicationGroup has been created are these boolean fields still controlled by the primary or made immutable?
There was a problem hiding this comment.
They're status-only and re-read from Describe each reconcile. Tested on a dev cluster: AWS rejects any TLS change on a member, so transitEncryptionEnabled is now datastore-owned on the ReplicationGroup side (5bf3425). Auth tokens can still be set or rotated on a TLS member, but DescribeGlobalReplicationGroups keeps reporting AuthTokenEnabled: false afterwards, so the GRG status reflects that; noted under known limitations.
Stop filtering read-path Spec values by whether the manifest declares them. The runtime writes sdkFind's Spec into the CR only on adoption or late-init, so the filter guarded nothing in normal reconciles and kept adoption from importing the datastore's configuration. modifyDelta now drops a difference on an undeclared inherited field instead, and an update with nothing modifiable returns desired rather than latest.
AWS rejects any in-transit encryption change on a Global Datastore member, so drop TransitEncryptionEnabled from a member's delta, adopt the observed value into spec, and strip TransitEncryptionEnabled/TransitEncryptionMode from the ModifyReplicationGroup request.
…owned AWS rejects resharding and durability changes on a Global Datastore member, and primaryClusterID can only change with automatic failover off, which a member cannot turn off. Adopt the observed shard count (refreshing the last-requested annotations so the resharding-rollback guard does not go Terminal) and durability, drop primaryClusterID from the delta, and strip Durability and PrimaryClusterId from the ModifyReplicationGroup request.
|
/retest |
| PrimaryReplicationGroupId: | ||
| is_immutable: true | ||
| references: | ||
| resource: ReplicationGroup | ||
| path: Spec.ReplicationGroupID |
There was a problem hiding this comment.
Was testing failovers on one of my clusters and found that this ref still names the original primary after a failover occurs. Since the field is immutable, it can't be corrected. Nothing flag that it no longer matches, and the only way to know which member is the primary is by checking status. It becomes a bigger problem if the ex-primary-now-secondary gets disassociated and deleted; the references can't resolve and this CR ends up stuck in ACK.ReferencesResolved=Unknown. Related: because the ref is resolved on every reconcile, it also fails whenever the primary RG CR isn't Synced, which includes during the GRG's own scale or upgrade putting the primary RG into modifying, so a ref-based GRG stops observing its own changes until the primary re-syncs.
Updating the spec to match the primary member in status would cause more problems than it would solves (the immutability rule rejects it, GitOps would drift after every failover, and the new primary may not be an ACK-managed RG). I think the best course of action would be to treat this as a create-time input only, and stop resolving it afterwards. We should also show the current datastore members from status.members in a printer column, to make the current primary/secondaries easy to see.
| PrimaryReplicationGroupId: | ||
| is_immutable: true | ||
| references: | ||
| resource: ReplicationGroup | ||
| path: Spec.ReplicationGroupID |
There was a problem hiding this comment.
Another issue with this field: when a GRG is adopted without the primaryReplicationGroupID/Ref in spec, the field is never imported. Nothing ever fills this field on read, as DescribeGlobalReplicationGroups has no primary-ID field, so the spec that adoption imports never gets it. This could be fixed by filling it on read from the primary member via a custom describe hook, but that would record the current primary rather than the one the datastore was created from. Another solution here is the same as my other comment: treat this field as create-time only. That way, both the ID-or-ref validation and the ref resolution only apply before the GRG exists. But that also means the field wouldn't get written into spec on adoption, and that's not ideal.
| var datastoreOwnedSpecPaths = []string{ | ||
| "Spec.CacheNodeType", | ||
| "Spec.CacheParameterGroupName", | ||
| "Spec.Engine", | ||
| "Spec.EngineVersion", | ||
| "Spec.AutomaticFailoverEnabled", | ||
| "Spec.TransitEncryptionEnabled", | ||
| "Spec.Durability", | ||
| "Spec.NumNodeGroups", | ||
| } |
There was a problem hiding this comment.
I'm realizing that these would cause GitOps drift if we write datastore-owned fields into spec. We've seen this kind of pattern before in this controller for other resources, so I think we should take it into consideration for this PR, as that back-and-forth with this many fields can cause a lot of churn. I wonder if there's a way to treat datastore members as compare: is_ignore, but only while they are members. We'd have to do an adoption if they're ever disassociated from the datastore though, since the spec would go stale pretty quickly.
There was a problem hiding this comment.
Could also have a documentation.yaml line to specify that datastore-owned fields in spec won't match status if they are updated through the datastore. Also look into adding an annotation for replication groups owned by datastores so we can ignore drift on those resources even after they've been disassociated with the datastore.
This is an alternative to writing back to spec that's causing the flip-flopping with GitOps tools. Also an alternative to the solution I mention in this comment, since that one-time adoption just get written back by GitOps.
…y at create ReplicationGroup: while the group is a Global Datastore member, drop the datastore-owned differences in modifyDelta instead of copying AWS values into spec, restore the declared values on update results, and skip the shard and failover paths that compare desired against latest directly. GlobalReplicationGroup: stop resolving primaryReplicationGroupRef once the datastore exists, fill primaryReplicationGroupID on adoption from the PRIMARY member, and add a PRIMARY printer column. Document both in a new documentation.yaml.
…-global-replication-group aws-controllers-k8s#245 makes autoMinorVersionUpgrade a ReplicationGroup spec field. AWS rejects it on a Global Datastore member, so it joins the datastore-owned set.
gobos12
left a comment
There was a problem hiding this comment.
After the GRG is created (or fails over, or is deleted), nothing reconciles the member RG CRs, so their status.globalReplicationGroupInfo, role and the GlobalDatastoreMember advisory stay stale until the next resync, which is 10h by default. Datastore-owned fields are still protected, because any spec edit re-reads membership from AWS. But the CR doesn't show it's a member, and a FieldExport of the role is wrong after a failover. Can we set requeue_on_success_seconds on ReplicationGroup so membership changes show up within a bounded interval? Something like 600s seems fine: it applies to every RG, not just members, but that's one extra read cycle per RG every 10 minutes, and operators can still override it per kind with --reconcile-resource-resync-seconds.
…y on updates Joining, leaving or failing over a Global Datastore happens outside the ReplicationGroup, so re-read it every 600s to keep globalReplicationGroupInfo and the member advisory current. Update results start from desired, whose conditions the runtime cleared, so set the advisory there too.
Issue #, if available: fixes aws-controllers-k8s/community#1048
Description of changes:
Adds the
GlobalReplicationGroupresource (ElastiCache Global Datastore), and makesReplicationGroupCRs work as datastore members. Supersedes #228; co-authored with @NicholasBlaskey, whose live testing established the API rules below.GlobalReplicationGroup
ModifyGlobalReplicationGrouptakes one logical change per request, so each reconcile applies one change (shard count via Increase/Decrease) and requeues until the delta is empty.ShowMemberInfo).cacheParameterGroupNameis an engine-upgrade input only: it is sent with an engine-family or major-version upgrade and never produces a delta.util.SyncTags); create-time tags come from the SDK bump in chore: bump aws-sdk-go-v2 to v1.47.4 #247.status.membersfollows failovers and members joining or leaving. ACustomUpdateConditionshook keeps Synced tied toavailable/primary-only, as sagemaker-controller does.primaryReplicationGroupRef/IDare create-time inputs. They are not resolved after the datastore exists, so a failover or an unsynced primary RG CR no longer blocks it, and they go stale after a failover. A newPRIMARYcolumn shows the current primary. On adoption the ID is filled from the current PRIMARY member.ReplicationGroup as a datastore member
AWS rejects changing node type, parameter group, engine, engine version, automatic failover, auto minor version upgrade, in-transit encryption, durability, shard count and
primaryClusterIDon a member, even to the current value (verified live on primary and secondary). Whilestatus.globalReplicationGroupInfonames a datastore:DBInstance/DBClusterhandling and Terraform's equivalent;ModifyReplicationGroup, and the shard and failover paths are skipped;ACK.Advisorycondition and the field docs say so. Once the group leaves, normal reconciliation resumes, so a stale node type or shard count fails withInvalidParameterCombinationand a lower engine version is rejected until the spec is updated.Membership changes happen outside the
ReplicationGroup, so everyReplicationGroupnow requeues every 600s;--reconcile-resource-resync-secondsstill overrides this per kind. The status-based Synced conditioncustomSetOutputalready sets is unaffected.Testing
Known limitations
adopt-or-createadoption policy needsprimaryReplicationGroupReforID.status.authTokenEnabledstaysfalseafter an auth token is added to a member, because that is what AWS reports.Excluded (pre-existing, separate follow-ups)
util.EngineVersionsMatchpanics on an empty version; guarded locally here.ReplicationGroup'ssetNodeGroupConfigurationnever populates the read-back slice, so a declarednodeGroupConfigurationalways shows a difference.ReplicationGroupupdates may write a pendingengine/transitEncryptionEnabledfrom the Modify response into spec (unverified).ReplicationGroup(globalReplicationGroupID): community#3099.Agent-assisted: authored by an AI agent (
backlog-burndown-bot) under human supervision — @gobos12By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.