[WIP] Node image builds for 3.10 by kwoodson · Pull Request #1224 · openshift/release

kwoodson · 2018-08-20T19:42:42Z

First attempt at node image build with e2e tests.

kwoodson · 2018-08-20T19:43:41Z

Currently failing with az commands not running in containers.

stevekuznetsov

This doesn't feel good. We are (ab)using the template mechanism to embed bash into pods and not really making any use of the ci-operator flows from what I can tell.

stevekuznetsov · 2018-08-22T15:48:35Z

We should be doing this with the tag specification in ci-operator for the 3.10 branch, not here

Agreed. I'm not sure I understand the tag specification completely. This made it very easy.

Hopefully the config reference is useful https://github.com/openshift/ci-operator/blob/master/CONFIGURATION.md#tag_specification

mjudeikis · 2018-08-23T08:18:41Z

It is still missing ci-operator config part in this PR too.

mjudeikis · 2018-08-30T17:11:35Z

s/azure/e2e-azure-secret/

This still needs changing :)

Yeah, I was still testing it locally. I thought this was necessary for local tests.

mjudeikis · 2018-09-05T10:34:35Z

What about RHEL images?

Today we have separate jobs for the rhel and centos. How do you foresee us having a single template and passing in variables for the build job? I can variable-ize the flavor of OS but how does that get passed into the job?

@Kargakis, any opinion on how this would work?

mjudeikis · 2018-09-05T10:34:55Z

still outstanding

mjudeikis · 2018-09-05T10:37:07Z

mjudeikis · 2018-09-05T10:38:10Z

What do you mean loops indefinitely? It shouldn't. What do you see if we dont have timeout?

I experienced failures where the cluster creation would wait indefinitely even though there were failures in the deployment. This occurred when I shared the same resource group with the node build and the cluster build. This pod would never complete so I added a timeout.

mjudeikis · 2018-09-05T10:43:28Z

You might want to update :

master.json	
release-3.10.json
release-3.11.json

to have similar format tag_specification to point to release of origin.

What do you want to do with prow config for this template? we would need still to add it. Separate PR?

0xmichalis · 2018-09-05T14:49:44Z

Why this change?

This is thing we didn't do, but we suppose to. This tags all particular release images into our ci-namespace for us to use. If we need ansible,origin, or any other image from another team - this is how we get them so we dont need to build it ourselves.

0xmichalis · 2018-09-05T14:50:22Z

Please remove the branched jobs, they don't apply to us (at least yet)

kwoodson · 2018-09-05T15:42:24Z

@Kargakis

Please remove the branched jobs, they don't apply to us (at least yet)

What did you mean by this? The release-* files? If that's the case, how do we get 3.10 builds since we need to specify a origin-v3.10 in the release-3.10.yml file under tag_specification?

On a side note, how do we get 3.10 builds for centos and rhel without hard coding them into the template? I put in a variable but I'm unsure how that variable gets populated for the actual run of the job.

0xmichalis · 2018-09-05T15:55:03Z

Your ci-operator jobs should be jobs we run in openshift-ansible and also periodically, right? The openshift-azure and azure-misc release jobs are not used.

mjudeikis · 2018-09-05T16:18:45Z

Your ci-operator jobs should be jobs we run in openshift-ansible and also periodically, right? The openshift-azure and azure-misc release jobs are not used.

we run then in openshift-azure but we consume artifacts from openshift-ansible runs. This way we dont need to rebuild ansible image each time.

kwoodson · 2018-09-05T16:26:39Z

2018/09/05 10:31:01 Running pod dev
2018/09/05 10:56:12 Container buildimage in pod dev completed successfully
2018/09/05 11:13:30 Container buildcluster in pod dev completed successfully
2018/09/05 11:14:22 Container tagimage in pod dev completed successfully
2018/09/05 11:14:59 Container teardown in pod dev completed successfully
2018/09/05 11:14:59 Pod dev succeeded after 43m59s
2018/09/05 11:14:59 Ran for 44m4s

0xmichalis · 2018-09-05T18:48:22Z

we run then in openshift-azure but we consume artifacts from openshift-ansible runs. This way we dont need to rebuild ansible image each time.

Not sure I follow. What's the relation between openshift-azure and the node image? What's the overhead of building the ansible docker image?

kwoodson · 2018-09-05T18:56:18Z

@Kargakis,

The node image is built using the origin-ansible container. Once it has completed, we will use the latest openshift-azure build to test a cluster build. Once the image has passed a cluster build, the image is tagged as valid: true. The teardown is called and the job completes. This job uses both openshift-ansible and openshift-azure to complete its image build and deploy of a cluster.

mjudeikis · 2018-09-05T18:59:09Z

Easier to show than to tell :) tomorrow will show this.

…

On Wed, 5 Sep 2018, 19:48 Michalis Kargakis, ***@***.***> wrote: we run then in openshift-azure but we consume artifacts from openshift-ansible runs. This way we dont need to rebuild ansible image each time. Not sure I follow. What's the relation between openshift-azure and the node image? What's the overhead of building the ansible docker image? — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADE4kEVOipD5bAKfnYYMeKWZM0z5ekhKks5uYBx3gaJpZM4WEl_J> .

0xmichalis · 2018-09-05T19:07:15Z

The node image is built using the origin-ansible container. Once it has completed, we will use the latest openshift-azure build to test a cluster build. Once the image has passed a cluster build, the image is tagged as valid: true. The teardown is called and the job completes. This job uses both openshift-ansible and openshift-azure to complete its image build and deploy of a cluster.

We have e2e tests for cluster deployments in openshift-azure that use the latest published image. We don't have e2e tests for gating openshift-ansible changes, right? Or at least we need to migrate those jobs away from Jenkins. We should still be able to pull openshift-azure and run a cluster deployment in a job that runs in openshift-ansible.

kwoodson · 2018-09-05T19:53:52Z

@Kargakis

We have e2e tests for cluster deployments in openshift-azure that use the latest published image.

Correct.

We don't have e2e tests for gating openshift-ansible changes, right? Or at least we need to migrate those jobs away from Jenkins.

This is the beginning of that work. The old node image jobs that are in Jenkins do not perform a cluster build. Currently we are not testing our images which is something I was hoping to fix by merging this PR. They used to test cluster builds but we removed that functionality when we stopped using acs-engine deployments.

We should still be able to pull openshift-azure and run a cluster deployment in a job that runs in openshift-ansible.

This is essentially the same thing. Whether we run the job from openshift-azure or openshift-ansible the need for each repo is equivalent. If you prefer moving the job to openshift-ansible and using openshift-azure for images that is fine. Currently the job is using openshift-azure's release-3.10.json and pulling the origin-ansible container image from openshift-ansible. This felt better as we own the process for openshift-azure and we don't have to touch openshift-ansible's files/config/templates.

0xmichalis · 2018-09-05T20:02:07Z

The problem with having this in openshift-azure is additional runtime in PR tests and extra churn to debug failures unrelated to the repo. If we want to test node builds we should be doing that in the repo the node build code resides (today openshift-ansible, maybe a separate repo in the future).

…

On Wed, Sep 5, 2018, 21:53 Kenny Woodson ***@***.***> wrote: @Kargakis <https://github.com/kargakis> We have e2e tests for cluster deployments in openshift-azure that use the latest published image. Correct. We don't have e2e tests for gating openshift-ansible changes, right? Or at least we need to migrate those jobs away from Jenkins. This is the beginning of that work. The old node image jobs that are in Jenkins do not perform a cluster build. Currently we are not testing our images which is something I was hoping to fix by merging this PR. They used to test cluster builds but we removed that functionality when we stopped using acs-engine deployments. We should still be able to pull openshift-azure and run a cluster deployment in a job that runs in openshift-ansible. This is essentially the same thing. Whether we run the job from openshift-azure or openshift-ansible the need for each repo is equivalent. If you prefer moving the job to openshift-ansible and using openshift-azure for images that is fine. Currently the job is using openshift-azure's release-3.10.json and pulling the origin-ansible container image from openshift-ansible. This felt better as we own the process for openshift-azure and we don't have to touch openshift-ansible's files/config/templates. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADuFf8beykcjLr-NEe-L74n4OTGnTJbkks5uYCvSgaJpZM4WEl_J> .

mjudeikis · 2018-09-11T16:47:08Z

I think this is not needed anymore?

0xmichalis · 2018-09-13T11:54:10Z

openshift-azure master is effectively origin 3.10 (we already use 3.10 images in master.yaml). I think this is the wrong place to branch node build jobs, and it's more appropriate to be using the openshift-ansible ci-operator specs.

0xmichalis · 2018-09-13T12:05:17Z

setup to mirror the install container in the other templates

changed to setup

0xmichalis · 2018-09-13T12:07:23Z

Is there any reason for a separate container for doing the tagging? It seems we would reduce the size of bash we maintain by moving this after running the node build playbook?

Sorry, just realized you need to tag after the e2e test succeeds. Nvmd.

0xmichalis · 2018-09-13T12:12:06Z

For ansible PRs and merged commits this needs to be ${LOCAL_IMAGE_BIN}. For periodic jobs, it should be fine as is. Unclear how to best template that.

0xmichalis · 2018-09-13T12:14:30Z

@stevekuznetsov @bbguimaraes any suggestions on how to clone a repo (openshift-azure) in an openshift-ansible test?

stevekuznetsov · 2018-09-13T17:03:12Z

@stevekuznetsov @bbguimaraes any suggestions on how to clone a repo (openshift-azure) in an openshift-ansible test?

What are you trying to acheive? This is the wrong flow.

I think this issue should have been tackled as "how does CI Operator support first-class creation and publishing of artifacts that are not container images or RPMs"

0xmichalis · 2018-09-13T22:29:34Z

The other templates we use for e2e tests in openshift-azure

…

On Thu, Sep 13, 2018, 21:23 Kenny Woodson ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In ci-operator/templates/build-azure-image-310.yaml <#1224 (comment)>: > + + source /etc/azure/credentials/secret + az login --service-principal -u ${AZURE_CLIENT_ID} -p ${AZURE_CLIENT_SECRET} --tenant ${AZURE_TENANT_ID} &>/dev/null + # TODO: Configure these from prow + export DNS_DOMAIN=osadev.cloud + export DNS_RESOURCEGROUP=dns + export DEPLOY_VERSION=v${OPENSHIFT_RELEASE} + export RUN_SYNC_LOCAL=true + export IMAGE_RESOURCEGROUP=images + + source /tmp/shared/environment + export IMAGE_RESOURCENAME=${IMAGE_NAME} + cd /go/src/github.com/openshift/openshift-azure/ + + # added timeout as this seems to loop infinitely awaiting cluster deployment + timeout 30m ./hack/create.sh ${RESOURCE_GROUP}-e2e Which other templates are you referring to? The other containers in this template? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADuFf-cQY7YvMBl-myH2NU7OoMIX8lpdks5uarCdgaJpZM4WEl_J> .

0xmichalis · 2018-09-13T22:37:03Z

Apart from the lack of support for publishing non-container image related artifacts in the ci-operator, the main issue here is that we have an openshift-ansible playbook for building cloud images and we need to run a cluster install on top of the image build using our own installer (openshift-azure). Ideally, this job should gate every openshift-ansible PR because the actual thing that is tested is the image build and the cluster install is there for verification but we can start with a periodic. Eventually I would love to see us move away from openshift-ansible altogether but we haven't discussed/prioritized that yet.

…

On Thu, Sep 13, 2018, 19:03 Steve Kuznetsov ***@***.***> wrote: @stevekuznetsov <https://github.com/stevekuznetsov> @bbguimaraes <https://github.com/bbguimaraes> any suggestions on how to clone a repo (openshift-azure) in an openshift-ansible test? What are you trying to acheive? This is the wrong flow. I think this issue should have been tackled as "how does CI Operator support first-class creation and publishing of artifacts that are not container images or RPMs" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADuFf-0qtecWWlpiUre01MIIZsvUeTRaks5uao_SgaJpZM4WEl_J> .

stevekuznetsov · 2018-09-14T01:43:03Z

As a short-term hack you could use extra_refs in the ProwJob. Please make a card on our JIRA to track publishing non-image artifacts for repos

0xmichalis · 2018-09-14T06:45:38Z

Please make a card on our JIRA to track publishing non-image artifacts for repos

https://jira.coreos.com/browse/DPTP-187

0xmichalis · 2018-09-14T07:13:09Z

@@ -0,0 +1,315 @@
+# This template is being used for testing OpenShift Ansible image builds for Azure.


This template needs to be moved in ci-operator/templates/openshift/openshift-azure.

0xmichalis · 2018-09-14T07:33:43Z

As a short-term hack you could use extra_refs in the ProwJob.

Is this going to work for a periodic job with skip_cloning: true?

0xmichalis · 2018-09-14T07:39:26Z

+
+          cp -r /tmp/shared/_data /go/src/github.com/openshift/openshift-azure/
+          cd /go/src/github.com/openshift/openshift-azure/
+          source /etc/azure/credentials/secret


You also need to source /tmp/shared/environment to get the resource group, right?

0xmichalis · 2018-09-14T07:39:45Z

+
+    # Performs cleanup of all created resources
+    - name: teardown
+      image: ${LOCAL_IMAGE_BIN}


LOCAL_IMAGE_SRC should be enough I think

BIN and SRC one difference is that BIN has generated code and binaries built already. We use binaries for sync image build in the local namespace. Didn't checked end-to-end but I think we need bin for this, as it is used for sync image build. Just to keep consistent.

But we don't care about building the sync image and we can just pull it from quay since what is tested here is the node build.

I guess a mismatch between the pushed image in quay and the plugin code that is pulled from git can fail the cluster install, assuming there is a breaking change in between.

our experience shows we good at this...

Probably a good one to have so we can stop breaking things :)

0xmichalis · 2018-09-14T07:39:59Z

+
+    # Runs an install
+    - name: setup
+      image: ${LOCAL_IMAGE_BIN}


LOCAL_IMAGE_SRC should be enough I think

0xmichalis · 2018-09-14T07:40:06Z

+        export DNS_DOMAIN=osadev.cloud
+        export DNS_RESOURCEGROUP=dns
+        export DEPLOY_VERSION=v${OPENSHIFT_RELEASE}
+        export RUN_SYNC_LOCAL=true


0xmichalis · 2018-09-14T07:40:22Z

+
+        source /etc/azure/credentials/secret
+        az login --service-principal -u ${AZURE_CLIENT_ID} -p ${AZURE_CLIENT_SECRET} --tenant ${AZURE_TENANT_ID} &>/dev/null
+        # TODO: Configure these from prow


0xmichalis · 2018-09-14T07:44:23Z

The periodic prowjob I came up with:

  - agent: kubernetes
    decorate: true
    name: azure-build-node-image-rhel-310
    extra_refs:
    - org: openshift
      repo: openshift-azure
      base_ref: master
    skip_cloning: true
    spec:
      containers:
      - args:
        - --secret-dir=/usr/local/e2e-azure-secret
        - --template=/usr/local/azure-image-build
        command:
        - ci-operator
        env:
        - name: CONFIG_SPEC
          valueFrom:
            configMapKeyRef:
              key: release-3.10.yaml
              # Use openshift-ansible ci-operator config for as long as
              # the node build code is part of that repo.
              name: ci-operator-openshift-openshift-ansible
        - name: JOB_NAME_SAFE
          value: azure-image-build
        - name: CLUSTER_TYPE
          value: azure
        image: ci-operator:latest
        name: test
        volumeMounts:
        - mountPath: /usr/local/azure-image-build
          name: job-definition
          subPath: azure-image-build.yaml
        - mountPath: /usr/local/e2e-azure-secret
          name: cluster-secrets-azure-file
      serviceAccountName: ci-operator
      volumes:
      - configMap:
          name: prow-job-azure-build-image
        name: job-definition
      - name: cluster-secrets-azure-file
        projected:
          sources:
          - secret:
              name: cluster-secrets-azure-file

Haven't actually tested but my question is, will LOCAL_IMAGE_SRC include the source code from extra_refs? @bbguimaraes @stevekuznetsov @petr-muller If so, @kwoodson I think we can get rid of the release-3.10.yaml ci-operator config that you are adding in this PR and use the openshift-ansible ci-operator config directly.

mjudeikis · 2018-09-14T08:38:20Z

I think we can get rid of the release-3.10.yaml ci-operator config that you are adding in this PR and use the openshift-ansible ci-operator config directly.

I think I know what you have in mind, but need more tech details on how it should be implemented.

0xmichalis · 2018-09-14T12:21:18Z

Closing in favor of #1502

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 20, 2018

kwoodson requested a review from mjudeikis August 20, 2018 19:42

stevekuznetsov reviewed Aug 22, 2018

View reviewed changes

kwoodson force-pushed the build_node_image branch from 59b63f7 to bc87d45 Compare August 30, 2018 16:28

mjudeikis reviewed Aug 30, 2018

View reviewed changes

kwoodson force-pushed the build_node_image branch 5 times, most recently from d284a98 to 3ec8a3a Compare August 31, 2018 21:34

mjudeikis reviewed Sep 5, 2018

View reviewed changes

Comment thread ci-operator/templates/build-azure-image-310.yaml Outdated

mjudeikis Sep 5, 2018

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

mjudeikis reviewed Sep 5, 2018

View reviewed changes

0xmichalis reviewed Sep 5, 2018

View reviewed changes

kwoodson force-pushed the build_node_image branch from 28d6765 to e007472 Compare September 10, 2018 13:55

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 10, 2018

mjudeikis reviewed Sep 11, 2018

View reviewed changes

0xmichalis reviewed Sep 13, 2018

View reviewed changes

Kenny Woodson added 3 commits September 13, 2018 15:26

Adding image builds.

35316a8

Adding versions to config and clean up.

3e40079

Minor clean up.

99d195c

kwoodson force-pushed the build_node_image branch from e007472 to 99d195c Compare September 13, 2018 19:26

Renamed build template to remove version.

2ed3d97

0xmichalis reviewed Sep 14, 2018

View reviewed changes

0xmichalis mentioned this pull request Sep 14, 2018

azure: add image node build job #1502

Merged

0xmichalis closed this Sep 14, 2018

		@@ -0,0 +1,315 @@
		# This template is being used for testing OpenShift Ansible image builds for Azure.

Uh oh!

Conversation

kwoodson commented Aug 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwoodson commented Aug 20, 2018

Uh oh!

stevekuznetsov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjudeikis commented Aug 23, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjudeikis commented Sep 5, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xmichalis commented Sep 5, 2018

Uh oh!

kwoodson commented Sep 5, 2018

Uh oh!

0xmichalis commented Sep 5, 2018

Uh oh!

mjudeikis commented Sep 5, 2018

Uh oh!

kwoodson commented Sep 5, 2018

Uh oh!

0xmichalis commented Sep 5, 2018

Uh oh!

kwoodson commented Sep 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjudeikis commented Sep 5, 2018 via email

Uh oh!

0xmichalis commented Sep 5, 2018

Uh oh!

kwoodson commented Sep 5, 2018

Uh oh!

0xmichalis commented Sep 5, 2018 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kwoodson commented Aug 20, 2018 •

edited

Loading

kwoodson commented Sep 5, 2018 •

edited

Loading