remove docker build in pre-merge [skip ci]

[pxLi]

Signed-off-by: Peixin Li pxli@nyu.edu

Move image build work to an independent image manage service, save some time for each build.

Tested in my forked repo. I will submit a verify PR against our repo after this one get merged.

[pxLi]

build

[NvTimLiu]

Looks good to me

Looks good to me, dev team please help to check

[pxLi]

@tgravescs @revans2 @jlowe

this could save around 10 - 15 mins for each run, but it will not do docker build even we have a change to the ubuntu Dockerfile (nightly test will help detect if any issue)

what we can do to improve for this PR is that if the PR include Dockerfile change, then we enable the docker build.
This could make the pre-merge pipeline complicated,

1. if ubuntu CI docker file change, enable docker build w/ a temp tag and push to Artifactory (to not conflict w/ other builds)
2. run test w/ the temp image
3. post action to clean the temp image in Artifactory

But changes to dockerfile is not that frequent, so to add extra steps to check and cleanup maybe not worthy for the 10 - 15 mins time saving, we may just keep what we are doing right now (docker build for every run).

Please kindly share your thoughts about this, thanks!

[jlowe]

IMO 10-15 minutes is a nice win on a build, and this win is for almost every premerge CI run. In terms of CI resource minutes saved per typical development day where there are many premerge CI runs, that's a sizeable savings overall.

I'm not a Jenkins/Blossom expert, but it seems like we could solve the need-to-build-Docker-sometimes problem by parameterizing the core premerge CI job with the Docker tag to use and then running the premerge flow like this:

1. premerge starts by launching a parent job that checks the branch to see if it modifies the Dockerfile
2. If Dockerfile modified then it launches the core premerge job with the standard Docker tag as a parameter
3. If Dockerfile not modified then:
4. it launches a sub-job to build the Dockerfile with a unique temporary tag
5. runs core premerge job with the temp Docker tag as a parameter
6. removes the temp Docker image after core premerge complete

There's minimal additional complexity for the core premerge job where there's more churn than the Docker image contents.
The parent job that handles the Dockerfile-modified checks and removes the temp Docker image if necessary seems relatively straightforward, and we already have most of the code needed for the sub-job to build the Docker image.

Thoughts? I'm totally fine if we want to hold off merging this change until the Dockerfile-modified logic is implemented so the premerge features remain intact and we don't temporarily lose the ability to check PRs that modify the Dockerfile.

[pxLi]

IMO 10-15 minutes is a nice win on a build, and this win is for almost every premerge CI run. In terms of CI resource minutes saved per typical development day where there are many premerge CI runs, that's a sizeable savings overall.

I'm not a Jenkins/Blossom expert, but it seems like we could solve the need-to-build-Docker-sometimes problem by parameterizing the core premerge CI job with the Docker tag to use and then running the premerge flow like this:

1. premerge starts by launching a parent job that checks the branch to see if it modifies the Dockerfile
2. If Dockerfile modified then it launches the core premerge job with the standard Docker tag as a parameter
3. If Dockerfile not modified then:
4. it launches a sub-job to build the Dockerfile with a unique temporary tag
5. runs core premerge job with the temp Docker tag as a parameter
6. removes the temp Docker image after core premerge complete

There's minimal additional complexity for the core premerge job where there's more churn than the Docker image contents.
The parent job that handles the Dockerfile-modified checks and removes the temp Docker image if necessary seems relatively straightforward, and we already have most of the code needed for the sub-job to build the Docker image.

Thoughts? I'm totally fine if we want to hold off merging this change until the Dockerfile-modified logic is implemented so the premerge features remain intact and we don't temporarily lose the ability to check PRs that modify the Dockerfile.

sounds OK. The reason we have complicated step for this is because its running on kubernetes, so we have to build the image in docker-in-docker container, and push the image to registry, then pull the image and start a new pod here for test running (we cannot just run the newly built image within the parent docker), at last we have to remove the temp image from registry to save space, but this is totally achievable.

I will try to add the Dockerfile-modified logic for this one later. Thanks!

[pxLi]

build

[pxLi]

build

[pxLi]

build

[pxLi]

Please let me merge this one and do a following verify PR to double check later, thanks!