Skip to content

Add ssh-bastion to interop jobs and exclude interop tests from FAR standalone jobs - #83179

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
ugreener:feat/interop-ssh-bastion
Aug 16, 2026
Merged

Add ssh-bastion to interop jobs and exclude interop tests from FAR standalone jobs#83179
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
ugreener:feat/interop-ssh-bastion

Conversation

@ugreener

@ugreener ugreener commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Problem

The NHC+FAR interop test (RHWA-1035) needs to stop kubelet on a worker node to simulate an unhealthy node for NHC to detect. Without ssh-bastion, the test relies on oc debug which consistently fails: the debug pod dies when kubelet stops, causing oc debug to exit with error. Kubelet then auto-restarts before NHC's 30s unhealthy duration threshold is met, so NHC never triggers remediation.

Additionally, the FAR standalone jobs (e2e-far-aws, e2e-far-weekly-aws) filter with !tier:upgrade but would still match interop tests. Since those jobs don't install NHC or deploy ssh-bastion, interop tests would fail.

Summary

  • Adds ssh-bastion step reference to both interop jobs, e2e-far-nhc-interop-aws (presubmit) and e2e-far-nhc-interop-weekly-aws (weekly periodic), matching the ssh-bastion usage in the standalone NHC jobs (e2e-nhc-aws, e2e-nhc-weekly-aws)
  • Excludes interop tests from FAR standalone jobs by changing the label filter from !tier:upgrade to !tier:upgrade && !tier:interop

Changes

  • medik8s-system-tests-main__4.22-konflux.yaml: added - ref: ssh-bastion to both interop jobs' test steps (e2e-far-nhc-interop-aws and e2e-far-nhc-interop-weekly-aws)
  • medik8s-system-tests-main__4.22-konflux.yaml: changed ECO_TEST_LABELS from !tier:upgrade to !tier:upgrade && !tier:interop in both e2e-far-aws (presubmit) and e2e-far-weekly-aws (periodic)

Summary by CodeRabbit

  • Updates the Medik8s OpenShift CI configuration for FAR AWS workflows.
  • Excludes interop-tier tests from FAR AWS and weekly FAR AWS ECO_TEST_LABELS.
  • Adds the ssh-bastion step to the FAR/NHC interop AWS workflow after AWS credential setup.
  • This allows CI to stop kubelet directly on worker nodes so NHC can reach its 30-second unhealthy threshold and remediate the node.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 35ccaffb-ae15-413c-897a-051c27901c1d

📥 Commits

Reviewing files that changed from the base of the PR and between 2337aba and f579be0.

📒 Files selected for processing (1)
  • ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml

Included review availability: Your plan includes up to 2 reviews per rolling hour; 1 remains after this review.


Walkthrough

The Medik8s FAR AWS workflows exclude upgrade and interop tiers from test selection. The FAR/NHC interop AWS workflows run ssh-bastion after AWS credential setup.

Changes

Medik8s AWS workflow updates

Layer / File(s) Summary
FAR AWS test selection
ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml
The FAR AWS and weekly FAR AWS workflows exclude upgrade and interop tiers from ECO_TEST_LABELS.
FAR/NHC interop AWS bastion step
ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml
The workflows run ssh-bastion after AWS credential setup.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to f579b

This localized configuration change adds the required SSH access step for interop jobs and excludes incompatible tests from standalone FAR jobs; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested labels: rehearsals-ack

Suggested reviewers: beekhof


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The PR adds two ssh-bastion refs; its bash -x deployment prints “The bastion address is ${bastion_host}” for the AWS LoadBalancer service, exposing infrastructure hostnames in logs. Mask or suppress the bastion address before CI log collection, while preserving sanitized error output.
✅ Passed checks (14 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The diff changes only YAML labels and static step references; it adds no Ginkgo title calls or dynamic test-title values.
Test Structure And Quality ✅ Passed The diff changes only CI YAML labels and step references; it introduces no Ginkgo It blocks or test setup, cleanup, waits, or assertions for this check to assess.
Microshift Test Compatibility ✅ Passed The diff changes only CI YAML labels and ssh-bastion references; it adds no Ginkgo tests or MicroShift-sensitive API, resource, or feature usage.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The HEAD^..HEAD diff changes only one CI YAML file: labels and ssh-bastion steps. It adds no Ginkgo e2e tests, so SNO compatibility checks do not apply.
Topology-Aware Scheduling Compatibility ✅ Passed The commit changes only one CI YAML file: test labels and an existing ssh-bastion step reference. It adds no deployment, operator, controller, or scheduling constraints.
Ote Binary Stdout Contract ✅ Passed The PR changes only one YAML CI configuration file; it introduces no OTE binary source or process-level stdout writes.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The diff changes only YAML job configuration: label filters and ssh-bastion references. It adds no Ginkgo e2e tests or IPv4/external connectivity assumptions.
No-Weak-Crypto ✅ Passed The diff only changes test labels and adds the existing ssh-bastion step; no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret comparisons were introduced.
Container-Privileges ✅ Passed The commit adds step references and label filters only; the changed config and ssh-bastion ref declare no privileged, host*, SYS_ADMIN, root, or allowPrivilegeEscalation settings.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes both main changes: adding ssh-bastion to interop jobs and excluding interop tests from FAR standalone jobs.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 10, 2026
@openshift-ci
openshift-ci Bot requested review from beekhof and razo7 August 10, 2026 13:49

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml`:
- Line 78: Update the ssh-bastion step, identified by the removed ref:
ssh-bastion, so its remote deploy script is fetched from a reviewed immutable
commit or otherwise ensure AWS credentials are unset before invoking it.
Preserve the step’s deployment behavior while preventing mutable master-branch
code from running with credentials.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 60a15060-e89e-4330-b789-6ae9e114ec07

📥 Commits

Reviewing files that changed from the base of the PR and between 9be95ee and d74a111.

📒 Files selected for processing (1)
  • ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml

@ugreener
ugreener force-pushed the feat/interop-ssh-bastion branch from d74a111 to b948f5b Compare August 10, 2026 14:11
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@ugreener ugreener changed the title Add ssh-bastion to FAR+NHC interop job for reliable kubelet stop Add ssh-bastion to interop job and exclude interop tests from FAR standalone jobs Aug 10, 2026
@ugreener

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@abrugaro

Copy link
Copy Markdown
Contributor

Do we need to add the ssh-bastion to the weekly interop job as well?

@abrugaro abrugaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci

openshift-ci Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

@abrugaro: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

test:
- chain: medik8s-setup
- ref: medik8s-aws-credentials
- ref: ssh-bastion

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing from weekly interop: ssh-bastion is added here for the presubmit interop job, but the parallel weekly periodic job (e2e-far-nhc-interop-weekly-aws, ~line 602) does not get it. Both jobs run the same interop tests (operator:far && operator:nhc && tier:interop) with NHC installed, and NHC tests need SSH bastion for kubelet-stop operations. Add - ref: ssh-bastion after medik8s-aws-credentials in the weekly interop job too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added - ref: ssh-bastion to e2e-far-nhc-interop-weekly-aws after medik8s-aws-credentials, matching the presubmit interop job e2e-far-nhc-interop-aws. The standalone NHC jobs (e2e-nhc-aws, e2e-nhc-weekly-aws) already use ssh-bastion for the same kubelet-stop reason. HEAD f579be0 (ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml:603).

@ugreener
ugreener force-pushed the feat/interop-ssh-bastion branch from b948f5b to f579be0 Compare August 16, 2026 10:57
@coderabbitai

coderabbitai Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@ugreener ugreener changed the title Add ssh-bastion to interop job and exclude interop tests from FAR standalone jobs Add ssh-bastion to interop jobs and exclude interop tests from FAR standalone jobs Aug 16, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@ugreener: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws medik8s/system-tests presubmit Ci-operator config changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-aws medik8s/system-tests presubmit Ci-operator config changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-weekly-aws N/A periodic Ci-operator config changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-weekly-aws N/A periodic Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@ugreener

Copy link
Copy Markdown
Contributor Author

@abrugaro Yes - it was needed and is now added in HEAD f579be0. e2e-far-nhc-interop-weekly-aws runs the same operator:far && operator:nhc && tier:interop selection with NHC, which needs the SSH bastion for kubelet-stop, so it now has - ref: ssh-bastion after medik8s-aws-credentials (ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml:603), matching the presubmit interop job. @razo7 raised the same point inline.

@razo7 razo7 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/pj-rehearse ack

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 16, 2026
@openshift-ci

openshift-ci Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abrugaro, razo7, ugreener

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ugreener

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-aws periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-nhc-interop-weekly-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@razo7

razo7 commented Aug 16, 2026

Copy link
Copy Markdown
Member

/pj-rehearse ack

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@razo7: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Aug 16, 2026
@openshift-ci

openshift-ci Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

@ugreener: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 56fa2dc into openshift:main Aug 16, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants