Skip to content

CNTRLPLANE-1617: add event-ttl to hypershift - #7202

Closed
tjungblu wants to merge 1 commit into
openshift:mainfrom
tjungblu:CNTRLPLANE-1617
Closed

tjungblu wants to merge 1 commit into
openshift:mainfrom
tjungblu:CNTRLPLANE-1617

Conversation

@tjungblu

@tjungblu tjungblu commented Nov 10, 2025 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

This implements what was proposed in openshift/enhancements#1857 - adding the event-ttl as a configurable value to the control plane.

This is done like #6019, as there is no KAS Operator in hypershift.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/CNTRLPLANE-1617

Special notes for your reviewer:

This has been entirely coded by Cursor, given the above PR as an example input.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 10, 2025
@openshift-ci-robot

openshift-ci-robot commented Nov 10, 2025 •

Copy link
Copy Markdown

@tjungblu: This pull request references CNTRLPLANE-1617 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

openshift-ci-robot commented Nov 10, 2025 •

Copy link
Copy Markdown

@tjungblu: This pull request references CNTRLPLANE-1617 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

[!NOTE]
Cursor Bugbot is generating a summary for commit 1056ae1. Configure here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Nov 10, 2025 •

Copy link
Copy Markdown
Contributor

Walkthrough

This PR introduces configurable Kubernetes API Server event TTL for HostedClusters. It adds a new constant, implements annotation-based parameter handling to convert minutes to duration format, updates config generation to use the dynamic value instead of hardcoded "3h", and propagates the annotation through controller reconciliation.

Changes

Cohort / File(s) Summary
Type and constant definitions
api/hypershift/v1beta1/hostedcluster_types.go
Adds new exported constant KubeAPIServerEventTTLMinutes documenting the annotation key for overriding event TTL with expected range (5-180 minutes).
KAS parameter handling
control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go, control-plane-operator/controllers/hostedcontrolplane/v2/kas/params_test.go
Adds EventTTL field to KubeAPIServerConfigParams struct with default "3h". Implements annotation parsing logic to convert minute strings to duration format (e.g., "180" → "180m") and populates EventTTL from KubeAPIServerEventTTLMinutes annotation when present. Tests updated to validate default and annotation-driven EventTTL behavior.
KAS config generation
control-plane-operator/controllers/hostedcontrolplane/v2/kas/config.go
Replaces hardcoded "3h" event-ttl value with dynamic p.EventTTL parameter in KubeAPIServerConfig.
Annotation propagation
hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go, hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go
Adds KubeAPIServerEventTTLMinutes annotation to the set of annotations mirrored from HostedCluster to HostedControlPlane during reconciliation. Test updated to include the annotation in both input and expected maps.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • Parameter parsing logic: Verify the minute-to-duration conversion correctly appends "m" suffix and handles edge cases within the 5-180 minute range specified in documentation.
  • Annotation propagation: Confirm the annotation is correctly threaded through the controller reconciliation flow and that the default value ("3h") is applied appropriately when the annotation is absent.
  • Test coverage: Ensure test cases adequately validate both default behavior and annotation-driven overrides.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.5.0)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci-robot

openshift-ci-robot commented Nov 10, 2025 •

Copy link
Copy Markdown

@tjungblu: This pull request references CNTRLPLANE-1617 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

[!NOTE]
Adds hypershift.openshift.io/event-ttl-minutes to override kube-apiserver --event-ttl, wired through KAS params/config and mirrored by the operator, with tests.

  • API:
    • Add annotation hypershift.openshift.io/event-ttl-minutes to configure kube-apiserver event TTL (minutes).
  • KAS (config/params):
    • Introduce defaultEventTTL (3h) and new EventTTL param; pass to --event-ttl instead of hardcoding.
    • Parse annotation value (minutes) and convert to duration string (e.g., 180 -> 180m).
  • Operator:
    • Mirror hypershift.openshift.io/event-ttl-minutes from HostedCluster to HostedControlPlane.
  • Tests:
    • Update/add unit tests to cover default and annotated EventTTL, and annotation mirroring.

Written by Cursor Bugbot for commit 1056ae1. This will update automatically on new commits. Configure here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
openshift-ci Bot requested review from jparrill and muraee November 10, 2025 11:06
@openshift-ci openshift-ci Bot added area/api Indicates the PR includes changes for the API area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Nov 10, 2025
if eventTTLMinutes := hcp.Annotations[hyperv1.KubeAPIServerEventTTLMinutes]; eventTTLMinutes != "" {
// Convert minutes to duration format (e.g., "180" -> "180m", "60" -> "60m")
kasConfig.EventTTL = fmt.Sprintf("%sm", eventTTLMinutes)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Unvalidated input threatens system configuration.

The EventTTL conversion from the annotation value doesn't validate that the input is a valid number or within the documented range of 5-180 minutes. Invalid values like "abc" or out-of-range values like "1" or "500" will be passed directly to the kube-apiserver configuration, potentially causing startup failures or configuration rejection.

Fix in Cursor Fix in Web

@openshift-ci-robot

openshift-ci-robot commented Nov 10, 2025 •

Copy link
Copy Markdown

@tjungblu: This pull request references CNTRLPLANE-1617 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📥 Commits

Reviewing files that changed from the base of the PR and between 4761146 and 1056ae1.

⛔ Files ignored due to path filters (1)
  • vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go is excluded by !vendor/**, !**/vendor/**
📒 Files selected for processing (6)
  • api/hypershift/v1beta1/hostedcluster_types.go (1 hunks)
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/config.go (1 hunks)
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go (3 hunks)
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/params_test.go (3 hunks)
  • hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go (1 hunks)
  • hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go (2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

  • hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/config.go
  • api/hypershift/v1beta1/hostedcluster_types.go
  • hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go
  • control-plane-operator/controllers/hostedcontrolplane/v2/kas/params_test.go
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
  • GitHub Check: Red Hat Konflux / control-plane-operator-main-on-pull-request
  • GitHub Check: Red Hat Konflux / hypershift-release-mce-211-on-pull-request
  • GitHub Check: Red Hat Konflux / hypershift-operator-main-on-pull-request
  • GitHub Check: Red Hat Konflux / hypershift-cli-mce-211-on-pull-request
🔇 Additional comments (5)
hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go (1)

2232-2232: LGTM! Annotation mirroring properly configured.

The addition of KubeAPIServerEventTTLMinutes to the mirrored annotations list is correct and follows the established pattern. This enables the event TTL configuration to flow from HostedCluster to HostedControlPlane, consistent with how other KAS annotations like KubeAPIServerGoAwayChance are handled.

control-plane-operator/controllers/hostedcontrolplane/v2/kas/params_test.go (1)

132-132: LGTM! Comprehensive test coverage for EventTTL feature.

The test changes properly validate the new EventTTL functionality:

  • Default value is correctly set (line 132)
  • Annotation-based override is tested with value "60" minutes → "60m" (lines 275-289)
  • Dedicated test case covers the annotation in isolation with value "180" minutes → "180m" (lines 291-306)

The tests follow established patterns and provide good coverage for both default behavior and annotation-driven configuration.

Also applies to: 275-306

hypershift-operator/controllers/hostedcluster/hostedcluster_controller_test.go (1)

685-685: LGTM: Test correctly verifies EventTTL annotation propagation.

The test properly validates that the KubeAPIServerEventTTLMinutes annotation with value "180" is mirrored from HostedCluster to HostedControlPlane during initial reconciliation.

Also applies to: 694-694

control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go (2)

25-25: LGTM: Default EventTTL value is reasonable.

The default of "3h" provides a sensible event retention period.


54-54: LGTM: EventTTL field appropriately added to config params.

Comment on lines +358 to +360
// KubeAPIServerEventTTLMinutes allows the --event-ttl parameter of the kube-apiserver to be overridden from its default of 3h (180 minutes)
// The value should be specified in minutes (e.g., "60", "180"). Valid range is 5-180 minutes.
KubeAPIServerEventTTLMinutes = "hypershift.openshift.io/event-ttl-minutes"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

LGTM! Verify range validation is enforced.

The constant definition is clear and follows the established annotation naming pattern. The documentation clearly specifies the expected format (minutes as numeric string) and valid range (5-180 minutes).

Please verify that the 5-180 minute range validation is enforced in the controller code (likely in params.go) that consumes this annotation:


🏁 Script executed:

#!/bin/bash
# Search for validation of the event-ttl-minutes annotation
rg -n -A 10 -B 2 'KubeAPIServerEventTTLMinutes|event-ttl-minutes' --type=go

Length of output: 13671



🏁 Script executed:

#!/bin/bash
# Search for validation rules or CEL rules that might validate the range
rg -n 'event-ttl|EventTTL' --type=yaml --type=go -g '!vendor/**' | head -50

Length of output: 33726


🏁 Script executed:

#!/bin/bash
# Look for validation in the HostedControlPlane CRD or validation code
rg -n 'validat.*event|CEL.*event|MinProperties|MaxProperties' --type=yaml -g '!vendor/**'

Length of output: 0


🏁 Script executed:

#!/bin/bash
# Search for any validation functions related to this annotation
ast-grep --pattern 'func $_($_, $_) { 
  $$$
  KubeAPIServerEventTTLMinutes
  $$$
}'

Length of output: 0


🏁 Script executed:

#!/bin/bash
# Check if there's validation in webhook or admission code
fd -e go -type f | xargs rg -l 'webhook|admission|validation' -g '!vendor/**' | xargs rg -n 'KubeAPIServerEventTTLMinutes|event-ttl' 2>/dev/null || echo "No webhook validation found"

Length of output: 32900


Add input validation for the event-ttl annotation range (5-180 minutes).

The annotation is documented to require a valid range of 5-180 minutes, but the implementation in control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go (lines 128-131) accepts any string value without validation. Invalid inputs like "1", "200", or "abc" would be passed directly to the kube-apiserver argument, potentially causing configuration errors at runtime.

Add range validation in the params.go code to enforce the documented 5-180 minute constraint before converting the annotation value to a duration string.

🤖 Prompt for AI Agents
In control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go around
lines 128-131, the code currently accepts the KubeAPIServerEventTTLMinutes
annotation string blindly; change it to parse the annotation as an integer (use
strconv.Atoi), validate that the parsed minutes are between 5 and 180 inclusive,
and only then convert to the kube-apiserver duration form (e.g.,
fmt.Sprintf("%dm", minutes)) and add the flag. If parsing fails or the value is
out of range, do not add the flag and emit a clear log or return an error
(consistent with surrounding error-handling), so invalid inputs like "1", "200",
or non-numeric values are rejected instead of passed through.

args.Set("etcd-prefix", "kubernetes.io")
args.Set("etcd-servers", p.EtcdURL)
args.Set("event-ttl", "3h")
args.Set("event-ttl", p.EventTTL)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

LGTM! Verify params.go handles conversion and defaults.

The change correctly uses the dynamic p.EventTTL parameter instead of a hardcoded value. The separation of concerns (config generation vs. parameter parsing/validation) is appropriate.

Please verify that the params.go file (referenced in dependencies) properly:

  1. Converts the minutes annotation value to duration format (e.g., "60" → "60m")
  2. Provides a default value (e.g., "3h") when the annotation is absent
  3. Validates the 5-180 minute range

🏁 Script executed:

#!/bin/bash
# Search for EventTTL parameter handling in params.go
rg -n -A 15 -B 5 'EventTTL|defaultEventTTL' control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go

Length of output: 3040


Range validation missing in EventTTL parameter handling.

The conversion and default value implementations are correct, but the code lacks validation for the 5-180 minute range. Line 130 in control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go directly converts the annotation value to duration format without bounds checking:

kasConfig.EventTTL = fmt.Sprintf("%sm", eventTTLMinutes)

Add validation to ensure eventTTLMinutes is between 5 and 180, otherwise invalid values will be passed to kube-apiserver and cause runtime failures.

🤖 Prompt for AI Agents
In control-plane-operator/controllers/hostedcontrolplane/v2/kas/config.go around
line 214, the EventTTL value is being set without validating that the minutes
value is within the required 5–180 minute range; add a range check on the parsed
eventTTLMinutes and if eventTTLMinutes < 5 or > 180, return an error (or
propagate an existing returned error) indicating the annotation is invalid
instead of calling args.Set, so invalid values are rejected before being passed
to kube-apiserver.

Comment on lines +127 to +131
kasConfig.EventTTL = defaultEventTTL
if eventTTLMinutes := hcp.Annotations[hyperv1.KubeAPIServerEventTTLMinutes]; eventTTLMinutes != "" {
// Convert minutes to duration format (e.g., "180" -> "180m", "60" -> "60m")
kasConfig.EventTTL = fmt.Sprintf("%sm", eventTTLMinutes)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Verify input validation for EventTTL annotation.

The annotation value is converted to duration format without validation. If the annotation contains non-numeric values (e.g., "abc") or already includes a unit suffix (e.g., "180m"), the resulting duration string would be invalid (e.g., "abcm" or "180mm").

While this pattern is consistent with other annotation handling in this file (e.g., MaxRequestsInflight), the additional transformation increases the risk of invalid configuration.

Run the following script to check if validation exists elsewhere:


🏁 Script executed:

#!/bin/bash
# Search for validation of KubeAPIServerEventTTLMinutes annotation
rg -n "KubeAPIServerEventTTLMinutes" -A 5 -B 5

# Search for validation or usage of EventTTL field
rg -n "\.EventTTL" -A 5 -B 5

# Look for duration parsing/validation in the KAS config code
rg -n "time\.ParseDuration|ParseDuration" --type=go -g "*/kas/*"

Length of output: 17624


Add input validation for EventTTL annotation value.

The annotation documentation specifies a valid range of 5-180 minutes, but the code doesn't validate this constraint. Invalid values like "abc", "180m", "1000", or "0" would produce invalid durations ("abcm", "180mm", "1000m", "0m") that fail at runtime when passed to kube-apiserver.

Add validation to verify:

  • The value is numeric
  • The value is within the documented range (5-180 minutes)

@openshift-ci-robot

openshift-ci-robot commented Nov 10, 2025 •

Copy link
Copy Markdown

@tjungblu: This pull request references CNTRLPLANE-1617 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

This implements what was proposed in openshift/enhancements#1857 - adding the event-ttl as a configurable value to the control plane.

This is done like #6019, as there is no KAS Operator in hypershift.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/CNTRLPLANE-1617

Special notes for your reviewer:

This has been entirely coded by Cursor, given the above PR as an example input.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 15, 2025
@openshift-merge-robot

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@jparrill jparrill left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

Thanks for the PR!, Some things:

  • The PR needs rebase
  • Looks good overall but I've mentioned some required changes to be done in the PR review


// KubeAPIServerEventTTLMinutes allows the --event-ttl parameter of the kube-apiserver to be overridden from its default of 3h (180 minutes)
// The value should be specified in minutes (e.g., "60", "180"). Valid range is 5-180 minutes.
KubeAPIServerEventTTLMinutes = "hypershift.openshift.io/event-ttl-minutes"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include the Annotation in the name

KubeAPIServerGoAwayChance = "hypershift.openshift.io/kube-apiserver-goaway-chance"

// KubeAPIServerEventTTLMinutes allows the --event-ttl parameter of the kube-apiserver to be overridden from its default of 3h (180 minutes)
// The value should be specified in minutes (e.g., "60", "180"). Valid range is 5-180 minutes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is the range validated?

},
},
{
name: "with event-ttl annotation",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add tests with range limits

@openshift-ci

openshift-ci Bot commented Jan 22, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill, tjungblu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 22, 2026
@openshift-ci

openshift-ci Bot commented May 11, 2026

Copy link
Copy Markdown
Contributor

@tjungblu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify 1056ae1 link true /test verify
ci/prow/e2e-aks 1056ae1 link true /test e2e-aks
ci/prow/e2e-aks-4-21 1056ae1 link true /test e2e-aks-4-21
ci/prow/e2e-aws-4-21 1056ae1 link true /test e2e-aws-4-21
ci/prow/unit 1056ae1 link true /test unit
ci/prow/e2e-azure-self-managed 1056ae1 link true /test e2e-azure-self-managed
ci/prow/verify-workflows 1056ae1 link true /test verify-workflows
ci/prow/security 1056ae1 link true /test security

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented May 12, 2026 •

Copy link
Copy Markdown
Contributor

Test Failure Analysis Complete

Job Information

  • Prow Jobs: pull-ci-openshift-hypershift-main-verify, pull-ci-openshift-hypershift-main-e2e-aks, pull-ci-openshift-hypershift-main-e2e-azure-self-managed, pull-ci-openshift-hypershift-main-security, tide
  • Build IDs: 1987839165181464576 (verify), 1987839147569582080 (e2e-aks), 2031415950808453120 (e2e-azure-self-managed), 2053898898140827648 (security)
  • PR: CNTRLPLANE-1617: add event-ttl to hypershift #7202 — CNTRLPLANE-1617: add event-ttl to hypershift
  • PR Head SHA: 1056ae1da8d050bc625f4da4475160db507a7522
  • PR Created: 2025-11-10 (6 months ago, never rebased)
  • GitHub Mergeable State: dirty (mergeable: false, rebaseable: false)

Test Failure Analysis

Error

Auto-merging api/hypershift/v1beta1/hostedcluster_types.go
CONFLICT (content): Merge conflict in api/hypershift/v1beta1/hostedcluster_types.go
Auto-merging control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go
Auto-merging hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
CONFLICT (content): Merge conflict in hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
Auto-merging vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go
CONFLICT (content): Merge conflict in vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go
Automatic merge failed; fix conflicts and then commit the result.
# Error: exit status 1

Summary

All 5 Prow job failures on PR #7202 are caused by git merge conflicts between the PR branch and the main branch. The PR was created on November 10, 2025 and its sole commit (1056ae1) has never been rebased. Over the past 6 months, main has diverged significantly — notably with the addition of KubeAPIServerServiceAccountTokenMaxExpiration annotation support, which was inserted into the exact same code locations where this PR adds KubeAPIServerEventTTLMinutes. Prow cannot merge the PR branch into main to build and test the code, so every job fails immediately during the clone/merge phase (within seconds). The tide controller also reports: "Not mergeable. PR has a merge conflict." No actual test, build, or security check was ever executed in the recent runs.

Root Cause

The PR branch (CNTRLPLANE-1617) has not been rebased or updated since its creation on November 10, 2025. The main branch has since received commits that modify the exact same code regions that this PR touches, creating 4 content-level merge conflicts:

  1. api/hypershift/v1beta1/hostedcluster_types.go — The PR inserts KubeAPIServerEventTTLMinutes after KubeAPIServerGoAwayChance (originally around line 357). On main, a new KubeAPIServerServiceAccountTokenMaxExpiration annotation was added in the same region (now lines 413–418), along with significant structural shifts from other features (Karpenter vCPUs, etcd snapshot restore, etc.).

  2. control-plane-operator/controllers/hostedcontrolplane/v2/kas/params.go — The PR adds EventTTL as a new field and associated logic immediately after the GoAwayChance section. On main, ServiceAccountTokenMaxExpiration was inserted in the same structural position (after GoAwayChance, line 127).

  3. hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go — The PR adds KubeAPIServerEventTTLMinutes to the mirroredAnnotations slice after KubeAPIServerGoAwayChance. On main, KubeAPIServerServiceAccountTokenMaxExpiration was added in the same position (line 2344), plus AWSKarpenterDefaultInstanceProfile was also inserted nearby.

  4. vendor/.../hostedcluster_types.go — Vendored copy of file Merge from openshift-hive/openshift #1, same conflict.

The original ci/prow/verify and ci/prow/e2e-aks runs from November 2025 ran for 27 and 112 minutes respectively, suggesting those were real test/build failures (not merge conflicts). However, their artifacts have been garbage-collected from GCS and cannot be analyzed. All subsequent job runs (March 2026 onward) fail within seconds due to merge conflicts.

Recommendations
  1. Rebase the PR branch onto current main — This is the only way to unblock CI. The author must:

    git fetch upstream main
    git rebase upstream/main

    Then resolve the 4 conflicts by placing the new EventTTL additions alongside the ServiceAccountTokenMaxExpiration code that was added on main in the same files.

  2. Update the vendored copy — After resolving api/hypershift/v1beta1/hostedcluster_types.go, run make vendor to sync vendor/github.com/openshift/hypershift/api/hypershift/v1beta1/hostedcluster_types.go.

  3. Re-run CI after rebase — Once conflicts are resolved and the branch is force-pushed, all 5 jobs will be automatically re-triggered by Prow against the updated merge base. This will finally reveal whether the PR's code changes themselves pass the verify, security, and e2e tests.

  4. Consider closing the PR if stale — This PR has been open for 6 months without updates. If the enhancement proposal (CNTRLPLANE-1575: Add support for event-ttl in Kube API Server Operator enhancements#1857) is still relevant, a fresh PR against the current main may be cleaner than resolving accumulated drift.

Evidence
Evidence Detail
GitHub merge state mergeable: false, mergeable_state: dirty, rebaseable: false
Tide status "Not mergeable. PR has a merge conflict." (state: error, since 2025-12-15)
e2e-azure-self-managed build-log.txt CONFLICT (content) in 4 files, Automatic merge failed, runtime: ~51 seconds
security build-log.txt Identical 4-file CONFLICT (content) pattern, runtime: ~27 seconds
verify/e2e-aks artifacts GCS artifacts garbage-collected (build IDs from Nov 2025); original runs took 27min/112min suggesting real failures at that time
PR last commit 1056ae1da8d050bc625f4da4475160db507a7522 — unchanged since Nov 10, 2025
Conflicting main addition KubeAPIServerServiceAccountTokenMaxExpiration annotation + params + controller mirroring — added to main in the same code regions where PR inserts KubeAPIServerEventTTLMinutes
Conflict file 1 api/hypershift/v1beta1/hostedcluster_types.go — PR line ~357 vs main line ~413
Conflict file 2 control-plane-operator/.../kas/params.go — both add after GoAwayChance
Conflict file 3 hypershift-operator/.../hostedcluster_controller.go — both add to mirroredAnnotations at same index
Conflict file 4 vendor/.../hostedcluster_types.go — vendored copy of file 1

@tjungblu

Copy link
Copy Markdown
Contributor Author

/close

@tjungblu tjungblu closed this May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/api Indicates the PR includes changes for the API area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants