Repository navigation
Feat/autominorupgrade - #245
ack-prow[bot] merged 1 commit into
Conversation
|
Hi @jvaibhav123-a11y. Thanks for your PR. I'm waiting for a aws-controllers-k8s member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
blacksoxx
left a comment
There was a problem hiding this comment.
Thanks for the PR, moving autoMinorVersionUpgrade into the spec is something I had hit as well. I checked out the head (571718b), built it and ran go test ./pkg/... with Go 1.25: everything passes, and the delta tests read well.
One question on the update path, since I was just bitten by the same mechanism on #243. The generated sdkUpdate now copies resp.ReplicationGroup.AutoMinorVersionUpgrade into ko.Spec after ModifyReplicationGroup (sdk.go around line 1285), and that ko is what the runtime patches back into the CR spec. If the modify response still carries the previous value for a moment, the spec would be flipped back to the old value and the next reconcile would send the change again. Did the live run cover the false to true patch and show the response already reporting true? If it does, nothing to do. If not, the existing pattern in CustomModifyReplicationGroupSetOutput (hooks.go, which keeps CacheNodeType and LogDeliveryConfigurations from desired) could cover this field too, and the e2e assertion on resource['spec']['autoMinorVersionUpgrade'] after the patch would be the place that catches it.
Everything else looks consistent to me: the is_read_only status copy set in customSetOutput, the delta removal when the spec leaves the field unset, and EngineVersionAutoUpgraded staying out of the way for a major change or a downgrade. Leaving the call on the status-plus-spec shape to the maintainers.
Yes, we checked with actual elasticcache cluster with flipping AutoMinorVersionUpgrade values true to false and vice versa. Its correctly reflecting in UI and inside status field of CR object |
|
Thanks for confirming the live flip in both directions, that answers it. Nothing further from my side. |
gobos12
left a comment
There was a problem hiding this comment.
Hi @jvaibhav123-a11y, thanks for the contribution! I tested this on one of my own clusters and left an in-line comment with my findings.
| // the service may have moved to a newer minor version than the one in spec; keep spec as is | ||
| if delta.DifferentAt("Spec.EngineVersion") && !delta.DifferentAt("Spec.Engine") && | ||
| desired.ko.Spec.EngineVersion != nil && latest.ko.Spec.EngineVersion != nil && | ||
| latest.ko.Spec.AutoMinorVersionUpgrade != nil && *latest.ko.Spec.AutoMinorVersionUpgrade && | ||
| util.EngineVersionAutoUpgraded(*desired.ko.Spec.EngineVersion, *latest.ko.Spec.EngineVersion) { | ||
| common.RemoveFromDelta(delta, "Spec.EngineVersion") | ||
| } |
There was a problem hiding this comment.
If a user sets autoMinorVersionUpgrade: false after an auto minor version upgrade moved the RG past the version in spec, the controller tries to downgrade the version on the next reconcile to match spec and the CR goes Terminal. I reproduced this on a live cluster, simulating the auto-upgrade with modify-replication-group --engine-version 8.2 while the spec stayed on 8.1:
- Spec 8.1, AWS upgraded to 8.2.0, AMVU true: Synced, no Modify
- Patched spec AMVU to false: one Modify, accepted
- Next resync, about a minute later: downgrade sent, CR Terminal with
InvalidParameterCombination: Cannot modify valkey from 8.2.0 to valkey version 8.1.0
I think the fix for this is to drop the AMVU condition from this skip (and in cache_cluster/delta_util.go). It should be safe given the only requests it would additionally suppress are downgrades to an older minor version on the same major, which Elasticache doesn't support anyway.
There was a problem hiding this comment.
@gobos12 Thanks a lot for your review and very good catch. I pushed the fix. Let me know if that looks good.
|
/ok-to-test |
|
MR shows in conflict status due version change. Do i need to run code against new code generator or? |
559626d to
f9e3bb5
Compare
|
@gobos12 @michaelhtm I have resolved conflicts and rebased. Please review |
gustavodiaz7722
left a comment
There was a problem hiding this comment.
Thanks for this, @jvaibhav123-a11y. I reviewed f9e3bb5. go build ./... and go test ./pkg/... pass, both on the head and when merged with #242. I also ran the controller on a dev EKS cluster against real ElastiCache. The spec field, the status copy and the engine version skip all behave as described. One problem in the update path needs a fix before this merges.
autoMinorVersionUpgrade is reverted when changed together with a field that uses a custom update path
CustomModifyReplicationGroup handles some changes itself instead of through the generated modify:
- engine version, engine, security group and parameter group changes go to
modifyReplicationGroup - replica and shard changes go to
increaseReplicaCount,decreaseReplicaCountandupdateShardConfiguration
None of those requests include AutoMinorVersionUpgrade. All of them build the returned resource with setReplicationGroupOutput, which now copies the response's old value into Spec.AutoMinorVersionUpgrade. The runtime then patches that spec onto the CR, so the user's change is overwritten and never sent.
Live run: Valkey, cache.t3.micro, starting at 8.1 with autoMinorVersionUpgrade: false, 60s resync. The request parameters below are from CloudTrail:
| Patch | Sent to ElastiCache | Result |
|---|---|---|
engineVersion: 8.2 and autoMinorVersionUpgrade: true |
ModifyReplicationGroup {engineVersion: 8.2} |
The controller patched the spec back to false about a second later. After the upgrade: Synced=True, false in AWS and in spec, stable across resyncs |
replicasPerNodeGroup: 1 and autoMinorVersionUpgrade: true |
IncreaseReplicaCount {newReplicaCount: 1} |
Spec patched back to false immediately |
autoMinorVersionUpgrade: true on its own |
ModifyReplicationGroup {autoMinorVersionUpgrade: true} |
true in spec, status and AWS, stable |
The last row is the path test_rg_update covers, which is why the e2e passes. It also answers @blacksoxx's question: the modify response reports the new value right away.
Suggested fix: send the field on the custom modify, and keep desired's value on the resource returned from the custom paths. A change that was skipped is then sent by the generated modify on the next reconcile. It's the same idea as CacheNodeType and UserGroupIDs in CustomModifyReplicationGroupSetOutput:
// newModifyReplicationGroupRequestPayload
if delta.DifferentAt("Spec.AutoMinorVersionUpgrade") && desired.ko.Spec.AutoMinorVersionUpgrade != nil {
input.AutoMinorVersionUpgrade = desired.ko.Spec.AutoMinorVersionUpgrade
}
// after setReplicationGroupOutput in modifyReplicationGroup, increaseReplicaCount,
// decreaseReplicaCount and updateShardConfiguration
out.ko.Spec.AutoMinorVersionUpgrade = desired.ko.Spec.AutoMinorVersionUpgradeA unit test that calls setReplicationGroupOutput with desired true and a response reporting false, then checks that newResourceDelta(desired, out) has no Spec.AutoMinorVersionUpgrade difference, fails on the current head and would cover this.
From reading the code, Description, SnapshotWindow and similar fields already have the same exposure on those paths, so the problem isn't new. The new field shouldn't add to it, though.
Confirmed working
- Create sends
autoMinorVersionUpgradefrom spec.status.autoMinorVersionUpgradematched the service value at every step. - With spec
engineVersion: 8.1and the group on 8.2, no downgrade is sent and the CR staysSynced=True. TurningautoMinorVersionUpgradeoff afterwards sent only that field, and nothing went Terminal over several resyncs. @gobos12's scenario is fixed.
Non-blocking
- The engine version skip no longer depends on
autoMinorVersionUpgrade. An explicit minor downgrade on a ReplicationGroup or CacheCluster now reportsSynced=Trueinstead of going Terminal, and the ReplicationGroup CR doesn't show the running version anywhere. That's reasonable since ElastiCache rejects the downgrade, but please say so in the PR description and release notes. The description still says the version is ignored "when auto upgrade is on" and doesn't mention the CacheCluster change. - In both
TestModifyDeltaEngineVersionWithAutoMinorVersionUpgradetests, theautoMinorVersionUpgradeinputs no longer affect the result. Rename the tests and drop that column, or keep one row that says the result doesn't depend on the setting. - On your earlier question about the conflict: #247 (aws-sdk-go-v2 v1.47.4) landed after your rebase and conflicts in
ack-generate-metadata.yaml. Rebase onto main and regenerate withmake build-controllerrather than editing the metadata file by hand, since the SDK bump can change generated code. - Please squash to a single commit with a descriptive title, e.g. "Add autoMinorVersionUpgrade to ReplicationGroup spec".
87b58c2 to
54ae182
Compare
@gustavodiaz7722 Thanks a lot for your review. I have made the changes based on above suggestions. Also tried to check scenarios on actual elastic cache cluster. Can you please review again? |
c0e923e to
36e055b
Compare
|
@gobos12 @michaelhtm @gustavodiaz7722 Can you please review MR and let me know if changes looks good to you. |
| // cluster mode, and other parameters. The resolved setting is reflected in | ||
| // the EffectiveDurability property of the replication group. For more information, | ||
| // see Durability (https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/durability.html). | ||
| // see Durability (http://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Durability.html). |
There was a problem hiding this comment.
Fixed it. The rebase or latest pull did not work.
i need explictly used with specific sdk version.
AWS_SDK_GO_VERSION=v1.47.4 \
make build-controller AWS_SERVICE=elasticache
| api_directory_checksum: f12346e97e05d0ca3b15b6eff8764e3b69e87bb9 | ||
| api_version: v1alpha1 | ||
| aws_sdk_go_version: v1.47.4 | ||
| aws_sdk_go_version: v1.41.10 |
There was a problem hiding this comment.
seems to be an issue with rebase and needs to be fixed
There was a problem hiding this comment.
Fixed it. The rebase or latest pull did not work.
i need explictly used with specific sdk version.
AWS_SDK_GO_VERSION=v1.47.4
make build-controller AWS_SERVICE=elasticache
36e055b to
8cac677
Compare
|
@gobos12 @michaelhtm @gustavodiaz7722 @kmala Sorry to trouble you again but can we have review again. Thanks a lot in advance for your review. |
|
Hi @gobos12 @michaelhtm @gustavodiaz7722 @kmala , could you please review this PR again? we would like to use this feature in our current implementation. Thanks in advance!!! |
gustavodiaz7722
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround, @jvaibhav123-a11y. I re-reviewed 8cac677.
The SDK regeneration is fixed: the metadata is back on v1.47.4, and against main the generated diff only adds autoMinorVersionUpgrade (the ReplicationGroup type, deepcopy, and the CRD plus its Helm copy). The branch predates the v1.11.1 release (#249) but merges cleanly, and go build ./... and go test ./pkg/... pass on the merged result.
Fix confirmed live
I deployed 8cac677 to a dev EKS cluster and repeated the two scenarios that failed last time. Same setup: Valkey, cache.t3.micro, starting at 8.1 with autoMinorVersionUpgrade: false, 60s resync. Request parameters are from CloudTrail:
| Patch | Sent to ElastiCache | Result |
|---|---|---|
engineVersion: 8.2 and autoMinorVersionUpgrade: true |
ModifyReplicationGroup {engineVersion: 8.2, autoMinorVersionUpgrade: true} |
Spec stayed true. After the upgrade: 8.2.0, true in spec, status and AWS, Synced=True, no further diffs across resyncs |
replicasPerNodeGroup: 1 and autoMinorVersionUpgrade: false |
IncreaseReplicaCount {newReplicaCount: 1}, then once the replica was available, ModifyReplicationGroup {autoMinorVersionUpgrade: false} |
Spec stayed false throughout. Ended with false in spec, status and AWS, Synced=True, no further diffs |
So the change is no longer lost on either path, and on the replica path it is applied by the next reconcile as intended.
One remaining change: the unit test
TestSetReplicationGroupOutputPreservesAutoMinorVersionUpgrade applies the override inside the test itself, so it passes even with the fix removed from hooks.go. Here is a replacement that calls the four update functions with an ElastiCache client stubbed through SDK middleware (no network), where every response still reports the old value. It passes on 8cac677 and fails on all four cases when I remove the fix.
To apply it in pkg/resource/replication_group/hooks_test.go:
- Delete
autoMinorVersionUpgradeDesiredandTestSetReplicationGroupOutputPreservesAutoMinorVersionUpgrade. That also puts the user group doc comment back aboveTestCustomModifyReplicationGroupSetOutputKeepsDesiredUserGroupIDs. - Add these imports. Both modules are already in
go.mod:"github.com/aws/smithy-go/middleware" ackmetrics "github.com/aws-controllers-k8s/runtime/pkg/metrics"
- Add the test below.
Replacement test
// stubSDKClient returns an ElastiCache client whose operations never reach the
// network: each call is answered by respond, and its input is recorded in calls
// under the operation name.
func stubSDKClient(calls map[string]interface{}, respond func(op string) interface{}) *svcsdk.Client {
return svcsdk.New(svcsdk.Options{
Region: "us-west-2",
Credentials: aws.AnonymousCredentials{},
APIOptions: []func(*middleware.Stack) error{
func(s *middleware.Stack) error {
return s.Initialize.Add(middleware.InitializeMiddlewareFunc("stub",
func(ctx context.Context, in middleware.InitializeInput, _ middleware.InitializeHandler) (
middleware.InitializeOutput, middleware.Metadata, error,
) {
op := middleware.GetOperationName(ctx)
calls[op] = in.Parameters
return middleware.InitializeOutput{Result: respond(op)}, middleware.Metadata{}, nil
}), middleware.Before)
},
},
})
}
// The custom update paths (ModifyReplicationGroup for engine changes, replica
// count and shard changes) return a replication group that still reports the
// previous AutoMinorVersionUpgrade. If that value reached the spec, the runtime
// would patch it onto the CR and a change made together with one of these
// fields would be lost.
func TestCustomUpdatePathsKeepDesiredAutoMinorVersionUpgrade(t *testing.T) {
staleRG := &svcsdktypes.ReplicationGroup{
ReplicationGroupId: aws.String("rg"),
Status: aws.String("modifying"),
AutoMinorVersionUpgrade: aws.Bool(false),
}
newPair := func() (*resource, *resource) {
latest := &resource{ko: &svcapitypes.ReplicationGroup{
Spec: svcapitypes.ReplicationGroupSpec{
ReplicationGroupID: aws.String("rg"),
Engine: aws.String("valkey"),
EngineVersion: aws.String("8.1"),
NumNodeGroups: aws.Int64(1),
ReplicasPerNodeGroup: aws.Int64(1),
AutoMinorVersionUpgrade: aws.Bool(false),
},
Status: svcapitypes.ReplicationGroupStatus{
Status: aws.String("available"),
NodeGroups: []*svcapitypes.NodeGroup{{
NodeGroupID: aws.String("0001"),
Status: aws.String("available"),
NodeGroupMembers: []*svcapitypes.NodeGroupMember{
{CacheClusterID: aws.String("rg-001")},
{CacheClusterID: aws.String("rg-002")},
},
}},
},
}}
desired := &resource{ko: latest.ko.DeepCopy()}
desired.ko.Spec.AutoMinorVersionUpgrade = aws.Bool(true)
return desired, latest
}
tests := []struct {
name string
op string
change func(desired *resource)
call func(rm *resourceManager, desired, latest *resource) (*resource, error)
}{
{
name: "engine version change",
op: "ModifyReplicationGroup",
change: func(d *resource) { d.ko.Spec.EngineVersion = aws.String("8.2") },
call: func(rm *resourceManager, d, l *resource) (*resource, error) {
return rm.modifyReplicationGroup(context.TODO(), d, l, newResourceDelta(d, l))
},
},
{
name: "replica count increase",
op: "IncreaseReplicaCount",
change: func(d *resource) { d.ko.Spec.ReplicasPerNodeGroup = aws.Int64(2) },
call: func(rm *resourceManager, d, l *resource) (*resource, error) {
return rm.increaseReplicaCount(context.TODO(), d, l)
},
},
{
name: "replica count decrease",
op: "DecreaseReplicaCount",
change: func(d *resource) { d.ko.Spec.ReplicasPerNodeGroup = aws.Int64(0) },
call: func(rm *resourceManager, d, l *resource) (*resource, error) {
return rm.decreaseReplicaCount(context.TODO(), d, l)
},
},
{
name: "shard count change",
op: "ModifyReplicationGroupShardConfiguration",
change: func(d *resource) { d.ko.Spec.NumNodeGroups = aws.Int64(2) },
call: func(rm *resourceManager, d, l *resource) (*resource, error) {
return rm.updateShardConfiguration(context.TODO(), d, l)
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
calls := map[string]interface{}{}
rm := &resourceManager{
metrics: ackmetrics.NewMetrics("elasticache"),
sdkapi: stubSDKClient(calls, func(op string) interface{} {
switch op {
case "DescribeCacheClusters":
return &svcsdk.DescribeCacheClustersOutput{CacheClusters: []svcsdktypes.CacheCluster{{
CacheClusterId: aws.String("rg-001"),
}}}
case "ModifyReplicationGroup":
return &svcsdk.ModifyReplicationGroupOutput{ReplicationGroup: staleRG}
case "IncreaseReplicaCount":
return &svcsdk.IncreaseReplicaCountOutput{ReplicationGroup: staleRG}
case "DecreaseReplicaCount":
return &svcsdk.DecreaseReplicaCountOutput{ReplicationGroup: staleRG}
case "ModifyReplicationGroupShardConfiguration":
return &svcsdk.ModifyReplicationGroupShardConfigurationOutput{ReplicationGroup: staleRG}
}
t.Fatalf("unexpected call to %s", op)
return nil
}),
}
desired, latest := newPair()
tt.change(desired)
out, err := tt.call(rm, desired, latest)
if err != nil || out == nil {
t.Fatalf("update returned (%v, %v)", out, err)
}
if _, ok := calls[tt.op]; !ok {
t.Fatalf("%s was not called, calls: %v", tt.op, calls)
}
if in, ok := calls["ModifyReplicationGroup"].(*svcsdk.ModifyReplicationGroupInput); ok {
if in.AutoMinorVersionUpgrade == nil || !*in.AutoMinorVersionUpgrade {
t.Errorf("ModifyReplicationGroup input AutoMinorVersionUpgrade = %v, want true", in.AutoMinorVersionUpgrade)
}
}
if newResourceDelta(desired, out).DifferentAt("Spec.AutoMinorVersionUpgrade") {
t.Errorf("returned Spec.AutoMinorVersionUpgrade = %v, want the desired true", *out.ko.Spec.AutoMinorVersionUpgrade)
}
})
}
}Once that's in, I'm happy to approve.
8cac677 to
20f8cf6
Compare
|
/label release/minor |
|
/LGTM |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gustavodiaz7722, jvaibhav123-a11y The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Thanks @gustavodiaz7722 for your review, guidance and approving PR. |
Releasing changes: * #245
…-global-replication-group aws-controllers-k8s#245 makes autoMinorVersionUpgrade a ReplicationGroup spec field. AWS rejects it on a Global Datastore member, so it joins the datastore-owned set.
Issue #, if available:
Description of changes:
What it does -
The field moves to spec, a read-only status copy is kept, and modifyDelta ignores a newer minor version when auto upgrade is on.
How it was tested
The unit tests and the updated test_rg_update e2e test.
Built image locally and executed change against aws elasticache
The generated-file diffs are expected (CRDs, deepcopy, ack-generate-metadata.yaml).
We have seen similar fix for RDS as well.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.