Skip to content

fix: compare ReplicationGroup log delivery with the observed state instead of an annotation - #242

Merged
ack-prow[bot] merged 5 commits into
aws-controllers-k8s:mainfrom
blacksoxx:replication-group-log-delivery-observed
Oct 26, 2026
Merged

ack-prow[bot] merged 5 commits into
aws-controllers-k8s:mainfrom
blacksoxx:replication-group-log-delivery-observed

Conversation

@blacksoxx

@blacksoxx blacksoxx commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available: aws-controllers-k8s/community#3024

Description of changes:

modifyDelta compared spec.logDeliveryConfigurations with the last request the controller had stored in the last-requested-log-delivery-configurations annotation, never with what the API reports. A resource that was adopted, or whose annotation was dropped by whatever applies its manifest, has no such annotation, so the comparison failed on every reconcile and ModifyReplicationGroup re-sent the same configuration each time, giving the desired resource state has changed and updated resource pair in the issue. The LogDeliveryConfigurations line in the diff on community#2711 is the same thing.

The comparison now runs against the observed state. A desired request is satisfied when status.logDeliveryConfigurations holds an entry for its log type with the same destination type, format and destination that is not being disabled, or when status.pendingModifiedValues.logDeliveryConfigurations holds that entry because a change is still in flight. A request with enabled: false is satisfied when no such entry exists or the entry is disabling. An entry in error with the right destination is left alone rather than re-requested every 30 seconds; changing the destination in the spec still re-requests it. A request without a log type is sent so the API rejects it with a terminal error, as before. The annotation and the two helpers that wrote and read it are removed since nothing reads them any more; stale annotations on existing resources are harmless.

Tests: TestLogDeliveryRequiresUpdate (14 rows, including the adopted case from the issue, a pending change, enabling, disabling, error and a Kinesis destination) and TestModifyDeltaLogDeliveryIgnoresAnnotation, which runs the generated delta without the annotation and fails on main. go build ./... and go test ./pkg/... pass; go vet on the package reports the same pre-existing unkeyed metav1.Time literals in the generated sdk.go as main. No e2e run: the existing suite has no log delivery case and I could not add one against a live account.

The e2e test_rg_log_delivery adds slow-log delivery to a CloudWatch log group, moves it to the second bootstrap group with another format, checks a resync leaves it alone, disables it and enables it again, asserting the CR status and the AWS side each time.

A request that leaves destinationType, logFormat or destinationDetails unset accepts what the API reports for them instead of re-sending. setAnnotationsFields removes the last-requested-log-delivery-configurations annotation that earlier controllers wrote. The log delivery e2e cleans up its group in a finally block.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…stead of an annotation

modifyDelta compared the desired log delivery requests with the last request stored in an annotation, so an adopted resource, which never carried it, re-sent the same configuration on every reconcile. The comparison now uses status.logDeliveryConfigurations and the pending values.

Signed-off-by: Youssef Omar Bouden <youssef.bouden2002@gmail.com>
@ack-prow
ack-prow Bot requested review from jlbutler and michaelhtm October 19, 2026 10:16
@ack-prow ack-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 19, 2026
@ack-prow

ack-prow Bot commented Oct 19, 2026

Copy link
Copy Markdown

Hi @blacksoxx. Thanks for your PR.

I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@knottnt knottnt assigned knottnt and gobos12 and unassigned knottnt Oct 20, 2026

@gobos12 gobos12 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @blacksoxx, thanks for the contribution! I ran this PR against a live cluster and confirmed it fixes aws-controllers-k8s/community#3024: an adopted ReplicationGroup with no annotation no longer re-sends log delivery configurations, and changing, disabling and re-enabling each produced exactly one ModifyReplicationGroup.

I noticed you mentioned the existing suite has no log delivery case and you couldn't add one against a live account. We used to have a test_rg_log_delivery case that was removed in #151, and service_bootstrap.py still provisions two CloudWatch log groups for it (LOG_GROUP/LOG_GROUP_2). I'd suggest restoring a version of that so we can properly test this e2e. The PR's e2e job would run it against the test account so you don't need your own, and I'll tag this PR ok-to-test so they run when you update the PR. Thanks!

@gobos12

gobos12 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@ack-prow ack-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 22, 2026
Adds slow-log delivery to a CloudWatch log group, moves it to the second bootstrap group with another format, checks a resync leaves it alone, disables it and enables it again, asserting the CR status and the AWS side each time.

Signed-off-by: Youssef Omar Bouden <youssef.bouden2002@gmail.com>
@blacksoxx

Copy link
Copy Markdown
Contributor Author

Thanks @gobos12 for the live confirmation and the pointer to the old test. 1ce7da1 brings back a test_rg_log_delivery built on the current fixtures: a Redis 7.1 group from replicationgroup_create_delete, slow-log delivery added to LOG_GROUP, moved to LOG_GROUP_2 with the text format, a resync window checked for no change, then disabled and enabled again, asserting the CR status and the DescribeReplicationGroups side at each step. The earlier e2e run died with a lost prow pod before the test started; the new commit should run it with the label.

The shared manifest substitutes engineVersion unquoted, so 7.1 was parsed as a number and the API server rejected the CR with a 422. 6.x, as the creation test uses, keeps slow-log support and parses as a string.

Signed-off-by: Youssef Omar Bouden <youssef.bouden2002@gmail.com>
@blacksoxx

Copy link
Copy Markdown
Contributor Author

The e2e run on 1ce7da1 failed before the test started: the shared replicationgroup_create_delete manifest substitutes engineVersion unquoted, so "7.1" reached the API server as a YAML number and the CR was rejected with a 422 (spec.engineVersion in body must be of type string); the other 26 tests passed. Fixed in the next commit by using 6.x like test_rg_creation_deletion does, which still supports slow-log delivery.

@gustavodiaz7722

Copy link
Copy Markdown
Member

Thanks @blacksoxx, and thanks for restoring the e2e. I went through the change and the e2e run on 748073a. The controller log for the test_rg_log_delivery CR shows exactly one updated resource per spec patch (enable, move, disable, re-enable) and none in the resync window, so the fix behaves as described. The log matters here because the resync assertion would pass even if the same config were re-sent: as the issue shows, a redundant ModifyReplicationGroup leaves the group synced. I also confirmed TestModifyDeltaLogDeliveryIgnoresAnnotation fails on the base commit.

Needs a rebase: #243 landed and also creates hooks_test.go, so there's an add/add conflict there and another in test_replicationgroup.py. Both look mechanical (merge the imports, keep both sets of tests).

A few non-blocking points:

  1. sameLogDeliveryDestination compares type, format and destination exactly. The annotation approach settled after one send whatever the API did with the request, but this one re-sends on every reconcile if the API defaults a field the user left unset. The API reference documents a default only for Enabled; LogFormat and DestinationType are optional with none listed. Could you confirm the API rejects a request without them, or skip the comparison when the desired field is nil?
  2. The pending-values branch didn't match during the log group/format move: 30s after the patch the controller still reported a diff against the old group in modifying, and the update was refused with Replication Group can not be modified, it is not in 'available' state. That's harmless, since nothing is sent until the group is available again, but the "requested change still pending" test row models a response we haven't seen live. Do you know what PendingModifiedValues.LogDeliveryConfigurations actually holds mid-modify?
  3. test_rg_log_delivery has no try/finally, so a failed assertion leaves the replication group behind. The durability tests in the same file do their cleanup in finally.
  4. Optional: existing CRs keep the old annotation forever, since getAnnotationsFields copies desired's annotations. Harmless, but setAnnotationsFields could delete it.

…og-delivery-observed

# Conflicts:
#	pkg/resource/replication_group/hooks_test.go
#	test/e2e/tests/test_replicationgroup.py
…n, clean up the e2e group

A request that leaves destinationType, logFormat or destinationDetails unset no longer re-sends on every reconcile because the API filled them in. setAnnotationsFields removes the last-requested-log-delivery-configurations annotation that older controllers wrote. The log delivery e2e deletes its group in a finally block like the durability tests.

Signed-off-by: Youssef Omar Bouden <youssef.bouden2002@gmail.com>
@blacksoxx

Copy link
Copy Markdown
Contributor Author

Thanks @gustavodiaz7722 for going through the controller log, the per-patch updated resource count is the confirmation I could not produce myself.

Main is merged in (0f16608, a merge rather than a rebase so the reviewed commits stay put; both sets of tests kept in hooks_test.go and test_replicationgroup.py), and e511b49 takes the four points:

  1. sameLogDeliveryDestination now skips any field the request leaves unset: destinationType, logFormat or destinationDetails as nil accept whatever the API reports (new row "type and format left unset accept what the API reports"). I could not confirm whether the API rejects a request without them, so the comparison no longer depends on it.
  2. The pending-values row models the PendingLogDeliveryConfiguration shape from the API reference (log type, destination type, details, format); I have not seen it populated live, and your run suggests it was not during the move. The branch is harmless when the list is empty (it falls through to the observed entry), but if you would rather not carry code for a response we have not observed I am happy to drop it.
  3. test_rg_log_delivery deletes its group in a finally block like the durability tests.
  4. setAnnotationsFields deletes the old last-requested-log-delivery-configurations annotation, with a unit test that checks the other annotations survive.

@gustavodiaz7722

Copy link
Copy Markdown
Member

/LGTM

@ack-prow ack-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 26, 2026
@ack-prow

ack-prow Bot commented Oct 26, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: blacksoxx, gustavodiaz7722

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ack-prow ack-prow Bot added the approved label Oct 26, 2026
@gustavodiaz7722

Copy link
Copy Markdown
Member

/label release/patch

@ack-prow ack-prow Bot added the release/patch Indicates this PR should trigger a patch version release on merge. label Oct 26, 2026
@ack-prow
ack-prow Bot merged commit 33204e8 into aws-controllers-k8s:main Oct 26, 2026
8 checks passed
backlog-burndown-bot added a commit to backlog-burndown-bot/elasticache-controller that referenced this pull request Oct 26, 2026
ack-prow Bot pushed a commit that referenced this pull request Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release/patch Indicates this PR should trigger a patch version release on merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants