🌱 increase ironic-image build timeout to 3600s #519
Conversation
Ironic-image build locally, with a fast disk, fast CPU, and fast net takes 500-600s, depending on pre-pulls, repository speeds etc. New cloud has none of those reliably, so it times out at 1000s far too often. Ironic image: 1000s -> 3600s (build time 500-600s) Sushy-tools: keep 1000s (build time 150s, 6x safety) VMBC: keep 1000s (build time 100s, 10x safety) Signed-off-by: Tuomo Tanskanen <tuomo.tanskanen@est.tech>
metal3-io-bot
added
the
size/XS
|
/override metal3-centos-e2e-integration-test-main metal3-ubuntu-e2e-integration-test-main |
|
@tuminoid: Overrode contexts on behalf of tuminoid: metal3-centos-e2e-integration-test-main, metal3-ubuntu-e2e-integration-test-main In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cc @kashifest @Rozzii |
|
/cherry-pick release-24.1 |
|
@tuminoid: once the present PR merges, I will cherry-pick it on top of release-24.1 in a new PR and assign it to you. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherry-pick release-24.0 |
|
@tuminoid: once the present PR merges, I will cherry-pick it on top of release-24.0 in a new PR and assign it to you. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cherry-pick release-23.1 |
|
@tuminoid: once the present PR merges, I will cherry-pick it on top of release-23.1 in a new PR and assign it to you. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/cc @NymanRobin |
There was a problem hiding this comment.
The change is good. However, please note that there have been some issues with VM scheduling for the jobs. For instance, this job took 15 hours when scheduling time is included: Build #62.
Maybe these jobs could be moved from Jenkins to github action runners in the future to improve reliability
|
/lgtm |
WOW that is not expected IMO, anyways yeah I would guess container image building could be a github action, I think that was considered originally too. |
Yes agreed. We can see that after scheduling, the build itself took almost 21minutes (~1250 sec), which would've timeouted as well, so this fix is needed. It would then leave 30mins for scheduling as it is counted as well. Similarly for sushy/vmbc, the safety allows ~10 minutes of scheduling. We should be able to accomplish that in Jenkins or with runners. Jenkins scheduling we can fix ourselves, but runners would be at mercy of CNCF and other runner users. edit: Correcting myself: for runners scheduling is not an issue, since if the workflow isn't scheduled, the time isn't running. Doh :) |
|
/cc @dtantsur @elfosardo |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dtantsur, kashifest, NymanRobin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
metal3-io-bot
added
the
approved
|
@tuminoid: new pull request created: #520 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@tuminoid: new pull request created: #521 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@tuminoid: new pull request created: #522 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
OCPBUGS-33335: Update Python Jinja2
Ironic-image build locally, with a fast disk, fast CPU, and fast net takes 500-600s, depending on pre-pulls, repository speeds etc. New cloud has none of those reliably, so it times out at 1000s far too often. Also scheduling time is considered when images are built in Jenkins, and often scheduling takes a long time. Thanks to #505 we get notifications of these failures now.
I will cherry-pick this to release branches as well.