Add support for NeMo SDK #131
Conversation
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
| interface: str = "eth0" | ||
| protocol: str = "tcp" | ||
| cpu_worker_memory_limit: str = "0" | ||
| rapids_no_initialize: str = "1" | ||
| cudf_spill: str = "1" | ||
| rmm_scheduler_pool_size: str = "1GB" | ||
| rmm_worker_pool_size: str = "72GiB" | ||
| libcudf_cufile_policy: str = "OFF" |
There was a problem hiding this comment.
Similar to my other comment, I often have trouble knowing what to set for these types of parameters. Is there anywhere the user might be able to refer to for recommendations of how to set these parameters for their specific cluster?
There was a problem hiding this comment.
I agree we should release a bigger guide on our recommendations for each parameter. For now I've included a docstring that should provide a bit more context. Let me know if you want me to change anything else to make it clearer.
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
| mkdir -p $LOGDIR | ||
| mkdir -p $PROFILESDIR |
There was a problem hiding this comment.
This is run inside the container ?
There was a problem hiding this comment.
Yes you're correct. It's a good call out, and it makes me think we maybe should've done this from the beginning since the contents of $LOGDIR and $PROFILESDIR get written inside the container, so they ought to be initialized in it too. Let me know if you disagree.
There was a problem hiding this comment.
So, i think because logs are accessed from outside the container so we should make it clear where the path is from outside the container, I think to make it clear we should echo $LOGDIR and $PROFILESDIR to help someone debugging this.
There was a problem hiding this comment.
Agreed, most of my setups typically mount these to be in a location that's accessible from outside the compute nodes so it would be good to keep in mind that the end goal for these logs/profiles is someplace that's within the mounted dirs
There was a problem hiding this comment.
Added echo and comments.
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
| exclusive=True, | ||
| time="04:00:00", | ||
| container_image="nvcr.io/nvidia/nemo:dev", | ||
| container_mounts=["/path/on/machine:/path/in/container"], |
There was a problem hiding this comment.
This is maybe a question for Nemo_sdk (apologies for my lack of familiarity). Can users pass in additional args here for other slurm options?
There was a problem hiding this comment.
Haha no need to apologize for your lack of familiarity. Yes the user can pass in additional args.
| mkdir -p $LOGDIR | ||
| mkdir -p $PROFILESDIR |
There was a problem hiding this comment.
Agreed, most of my setups typically mount these to be in a location that's accessible from outside the compute nodes so it would be good to keep in mind that the end goal for these logs/profiles is someplace that's within the mounted dirs
|
|
||
|
|
||
| @dataclass | ||
| class SlurmJobConfig: |
There was a problem hiding this comment.
Also tagging @jacobtomlinson who's done a lot of work on the dask/dask-cuda clusters with Slurm (among other things).
For now this mimics the command line setup to start clusters, but feel free to share any opinions you might have since this overlaps a lot with the dask-runners/dask-jobqueue api.
There was a problem hiding this comment.
Thanks @ayushdg. This seems to follow the common pattern that a lot of Slurm implementations use so I don't have any particular comments. I'm always keen to see how we can reuse code though, so maybe we could work towards a common base in dask-jobqueue that projects like this can use instead of reinventing it each time.
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
* Begin docs Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add slurm sdk example Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Use safe import Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix bugs in sdk Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Update docs and tweak scripts Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add interface helper function Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Update docs Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Fix formatting Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Add config docstring Signed-off-by: Ryan Wolf <rywolf@nvidia.com> * Address comments Signed-off-by: Ryan Wolf <rywolf@nvidia.com> --------- Signed-off-by: Ryan Wolf <rywolf@nvidia.com>
Description
NeMo SDK is a library designed to make running different parts of the NeMo FW easier across computing platforms. It serves as an enhanced version of the NeMo Framework Launcher. This PR adds an example and simple config shortcut to run NeMo Curator scripts on Slurm clusters using NeMo SDK.
Usage
See
examples/nemo_sdk/slurm.pyfor example usage.Checklist