Migrate doomsday pipeline to python #997
Conversation
|
Tested the command locally and its working as expected. Once the review changes are in, we can test it on prod |
| major_minor = ".".join(self.version.split(".")[:2]) | ||
| path = f"{self.workdir}/{major_minor}/{self.version}/{arch}" | ||
| mirror_cmd = f"oc adm release mirror quay.io/openshift-release-dev/ocp-release:{self.version}-{arch} --keep-manifest-list --to-dir={path}" | ||
| aws_cmd = f"aws s3 sync {path} s3://ocp-doomsday-registry/release-image/{path}" |
There was a problem hiding this comment.
Should workdir be part of the bucket?
There was a problem hiding this comment.
cmd_assert accepts both a string and a list as command. Think list is preferred. This allows python to do the shell escaping for us.
mirror_cmd = ["oc', 'adm', 'release', 'mirror',
f'quay.io/openshift-release-dev/ocp-release:{self.version}-{arch}',
'--keep-manifest-list',
'--to-dir={path}']
There was a problem hiding this comment.
Should workdir be part of the bucket?
Thanks for pointing that out, complete missed it. Should be fixed now
cmd_assert accepts both a string and a list as command. Think list is preferred. This allows python to do the shell escaping for us.
Also fixed
|
|
||
| major_minor = ".".join(self.version.split(".")[:2]) | ||
| path = f"{self.workdir}/{major_minor}/{self.version}/{arch}" | ||
| mirror_cmd = f"oc adm release mirror quay.io/openshift-release-dev/ocp-release:{self.version}-{arch} --keep-manifest-list --to-dir={path}" |
There was a problem hiding this comment.
Not part of your porting work but noticing there are more "multi" entries that might need backup?
skopeo list-tags docker://quay.io/openshift-release-dev/ocp-release | grep '4\.16\.13'
"4.16.13-aarch64",
"4.16.13-multi",
"4.16.13-multi-aarch64",
"4.16.13-multi-ppc64le",
"4.16.13-multi-s390x",
"4.16.13-multi-x86_64",
"4.16.13-ppc64le",
"4.16.13-s390x",
"4.16.13-x86_64",
Is 4.16.13-multi-x86_64 implicitly backed up when 4.16.13-multi is treated?
There was a problem hiding this comment.
@adobes1 lets keep this as a follow up. I'll give you read only access to the AWS cluster, once this PR is merged, make sure that files are being uploaded correctly to the bucket, this week.
|
|
||
| try: | ||
| if self.runtime.dry_run: | ||
| self.runtime.logger.info("[DRY RUN] Would have run %s", mirror_cmd) |
There was a problem hiding this comment.
"copy to local working directory" should be ok in dry-run scenario.
There was a problem hiding this comment.
Makes sense, I changed it
adobes1
force-pushed
the
migrate-doomsday
branch
from
6eae648
to
f7af28b
Compare
ashwindasr
force-pushed
the
migrate-doomsday
branch
from
f7af28b
to
09be81c
Compare
|
rebased to pull in new changes |
ashwindasr
force-pushed
the
migrate-doomsday
branch
2 times, most recently
from
fe0fe42
to
60fc7b3
Compare
ashwindasr
force-pushed
the
migrate-doomsday
branch
from
60fc7b3
to
cd6876c
Compare
|
@adobes1: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This reverts commit cd6876c.
|
The uv installation seems to be causing an issue. In openshift, we rely on the gobal pip packages, but uv uses a local venv. Lets figure it out in a following PR, and merge with pip3 intall for now |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ashwindasr The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
Migrate doomsday pipeline to Python and add retries to running
oc adm release mirror ...andaws s3 sync ...commands