WIP: Composer Jenkinsfile #82
Merged
Commits on Nov 12, 2021
Commits on Nov 13, 2021
Commits on Nov 15, 2021
Commits on Nov 16, 2021
-
-
-
Merge branch 'ravi/jenkinsfile' of github.com:mosaicml/composer into …
…ravi/jenkinsfile
-
-
-
-
-
-
-
-
-
-
-
-
Commits on Nov 19, 2021
Commits on Dec 3, 2021
Commits on Dec 21, 2021
Commits on Dec 22, 2021
-
Configure a default virtualenv in the dockerfile
`pip install -e` does not properly install console_scripts outside of a virtualenv In addition, a virtualenv elimiantes the need to use upgrade-alternatives. This also fixes apt, which can continue to use system python.
-
-
-
-
-
-
-
Added `--without-pip` since pip comes from system python. Setuptools breaks when using `--system-site-packages` without `--without-pip`
-
Need to install with pip (so pip is relative to the installation directory), but immediately upgrade it since the default setuptools is broken.
-
-
-
-
Including `--system-site-packages` with the `--upgrade` command. Otherwise it reverts to not including system packages.
-
-
-
-
-
-
-
-
Commits on Dec 27, 2021
-
Make one global virtualenv. Works in both user mode and root mode. Compatible if the user overrides it with their own virtualenv.
-
Make one global virtualenv. Works in both user mode and root mode. Compatible if the user overrides it with their own virtualenv.
-
-
-
-
Merge branch 'ravi/composer_in_virtualenv' of github.com:mosaicml/com…
…poser into ravi/composer_in_virtualenv
-
Commits on Jan 3, 2022
Commits on Jan 6, 2022
Commits on Jan 7, 2022
Commits on Jan 10, 2022
-
-
-
-
-
-
-
-
Running deepspeed tests via jenkins
Fixed ddp test incorreclty marked as gpu when it should be marked as deepspeed
-
-
-
-
Commits on Jan 11, 2022
-
Make the run directory rank-local; fix checkpoints saving and restoring
- Sharding the of the run directory accross ranks won't work in multi-node training. This change makes the run directory rank-local - Fixed callbacks and loggers to support rank-local run directories. Specifically, wandb and the run directory uploader now run on all ranks, not just rank zero - When using deepspeed with zero-1+, each rank writes to the checkpoint folder. Previously, only rank zero's data was being stored. Now, each rank's data is being stored by the rank-local run directory uploader. The checkpoint loader takes a checkpoint path that is parameterized by the rank, so each node will load only the shards of the checkpoint that is needed.
-
-
-
Merge branch 'ravi/rank_local_run_directory' of github.com:mosaicml/c…
…omposer into ravi/rank_local_run_directory
-
-
-
-
-
-
-
-
-
Commits on Jan 12, 2022
Commits on Jan 14, 2022
Commits on Jan 19, 2022
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.