Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] Use local caches on self-hosted runners, and code refactoring. #5846

Merged
merged 1 commit into from
Sep 20, 2022
Merged

[ci] Use local caches on self-hosted runners, and code refactoring. #5846

merged 1 commit into from
Sep 20, 2022

Conversation

feisuzhu
Copy link
Contributor

@feisuzhu feisuzhu commented Aug 23, 2022 •
edited
Loading

This PR starts as an attempt to remove actions/cache@v2 usage on self-hosted runners, but in the end a lot of refactoring is done here.

Things resolved in this PR:

  1. Use local caches on self-hosted runners (of course)
  2. Unified Windows CPU & GPU task build scripts, split build and testing process, major code refactoring.
  3. Mechanism for probing available resource to determine maximum parallelism (Windows + CPU tasks only for now)
  4. Caching pip download and generated virtual environment
  5. Extract docker related logic to ci-docker-run fucntion, simpifying yaml file and build scripts.

@feisuzhu feisuzhu added the ci label Aug 23, 2022
@netlify
Copy link

netlify bot commented Aug 23, 2022 •
edited
Loading

✅ Deploy Preview for docsite-preview canceled.

Name Link
🔨 Latest commit e34a35f
🔍 Latest deploy log https://app.netlify.com/sites/docsite-preview/deploys/6327e3c2300aed00095bcbed

@feisuzhu feisuzhu force-pushed the feat/local-sccache branch 8 times, most recently from 5f1e5d6 to 95aaed8 Compare August 30, 2022 06:36
@feisuzhu feisuzhu force-pushed the feat/local-sccache branch 6 times, most recently from 7776410 to 302c4a9 Compare September 6, 2022 03:53
@feisuzhu
Copy link
Contributor Author

feisuzhu commented Sep 6, 2022

/rerun

@feisuzhu feisuzhu force-pushed the feat/local-sccache branch 5 times, most recently from 466462a to 11e8fac Compare September 8, 2022 07:48
@feisuzhu feisuzhu force-pushed the feat/local-sccache branch 7 times, most recently from 673827f to 516d031 Compare September 15, 2022 04:17
@feisuzhu feisuzhu added the full-ci Run complete set of CI tests label Sep 16, 2022
@feisuzhu feisuzhu force-pushed the feat/local-sccache branch 17 times, most recently from 4ddcb83 to 4de840f Compare September 18, 2022 11:44
@feisuzhu feisuzhu changed the title [ci] Use local caches on self-hosted runners [ci] Use local caches on self-hosted runners, and code refactoring. Sep 19, 2022
ailzhang
ailzhang approved these changes Sep 19, 2022
View reviewed changes
Copy link
Contributor

@ailzhang ailzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a ton!!
nit: as discussed offline let's create an issue to track CI/CD pipeline overhaul, it'd be nice to have two images with "existing" and "target" workflow pipeline there ;)

@ailzhang
Copy link
Contributor

Also let's make sure this PR only gets merged after v1.1.3 branch cut. :P cc: @strongoier

@ailzhang ailzhang mentioned this pull request Sep 19, 2022
Merged
@feisuzhu feisuzhu marked this pull request as ready for review September 20, 2022 06:24
@feisuzhu feisuzhu merged commit b628d33 into taichi-dev:master Sep 20, 2022
@feisuzhu feisuzhu deleted the feat/local-sccache branch September 20, 2022 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ailzhang ailzhang ailzhang approved these changes

@jim19930609 jim19930609 jim19930609 approved these changes

Assignees
No one assigned
Labels
ci full-ci Run complete set of CI tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@feisuzhu @ailzhang @jim19930609