Adding an example for executing NeMo modules using kubernetes Python … #148

dpadmanabhan03 · 2024-07-09T16:09:37Z

…client

Description

This PR adds an example demonstrating the usage of the Kubernetes Python client to execute NeMo modules on the scheduler pod.

Usage

Updated documentation.
# Add snippet demonstrating usage

Checklist

I am familiar with the Contributing Guide.
New or Existing tests cover these changes.
The documentation is up to date with these changes.

ryantwolf

Thanks for adding this! The main thing is that the documentation could be improved a bit by explaining why each step is necessary and why your method overall would be preferred to the other existing options. Code looks good, short and sweet!

docs/user-guide/kubernetescurator.rst

ryantwolf

Most of my comments stem from my lack of familiarity with AWS EKS. I do think it's relevant to explain some of the "basics" of EKS since many of our users will be like me with little background in these systems. Thanks again for making this guide!

docs/user-guide/aws-examples/eks.rst

ryantwolf · 2024-07-19T18:43:03Z

docs/user-guide/aws-examples/eks.rst

+Running NeMo Curator on AWS EKS
+======================================
+
+Prerequisuites:


Typo. Should be "Prerequisites"

ryantwolf · 2024-07-19T18:43:41Z

docs/user-guide/aws-examples/eks.rst

+Prerequisuites:
+
+* EKS Cluster:
+    * `Dask Operator <https://kubernetes.dask.org/en/latest/operator_installation.html>`__


"Dask Operator" is included twice, and the link does not go anywhere either. Please remove one of the mentions and fix the link.

ryantwolf · 2024-07-19T18:51:27Z

docs/user-guide/aws-examples/eks.rst

+Running NeMo Curator on AWS EKS
+======================================
+
+Prerequisuites:


Could you please add a high level "Background" section at the top of this document? It should answer the following questions

What is AWS EKS at a high level?

What is this guide going to show me to do on AWS EKS with NeMo Curator, and what are the fundamental assumptions that it makes?

Will it cover how to run any prebuilt CLI script in NeMo Curator?

Will it cover how to run a python script that I have made?

Will it cover how to upload my data to EKS?

Is there any reason to prefer running NeMo Curator on AWS EKS?

The answer to this might be "only if you are familiar with it or if it's the only compute option you have" which is fine. Just wondering if it gives any benefits.

ryantwolf · 2024-07-19T18:53:41Z

docs/user-guide/aws-examples/eks.rst

+
+* EKS Cluster:
+    * `Dask Operator <https://kubernetes.dask.org/en/latest/operator_installation.html>`__
+    * If self managed node group is created with ubuntu worker nodes then install GPU operator.


If self managed node group is created with ubuntu worker nodes

If EKS managed node group is created with Amazon Linux 2 worker nodes

What determines if either of these conditions are true? Is it up to the user? If so, which one should we recommend the user do?

ryantwolf · 2024-07-19T20:35:12Z

docs/user-guide/aws-examples/eks.rst

+- Binds the pod-exec ClusterRole to a specific ServiceAccount (default in the default namespace).
+- This means that any pods using the default ServiceAccount in the default namespace will have the permissions specified in the pod-exec ClusterRole.
+
+ Now, we can spin up a client pod.


Delete the space at the beginning of this line.

ryantwolf · 2024-07-19T20:35:35Z

docs/user-guide/aws-examples/eks.rst

+          command: ["sh", "-c", "pip install kubernetes && sleep infinity"]
+    EOF
+
+Here, we are using a light-weight public python docker image and installing kubernetes Python client package so that we can run kubeclient.py from this client pod and connect to the scheduler pod to run existing NeMo modules.


Nit: "NeMo" -> "NeMo Curator"

ryantwolf · 2024-07-19T20:36:03Z

docs/user-guide/aws-examples/eks.rst

+    kubectl exec client-pod -- python3 kubeclient.py --command "add_id --scheduler-address localhost:8786 --input-data-dir=/nemo-workspace/arxiv --output-data-dir=/nemo-workspace/arxiv-addid/"
+
+
+This approach is allows the execution of NeMo modules within the scheduler pod from a separate client pod. This separation ensures that the client pod can be provisioned with specific permissions tailored for executing commands and accessing resources within the Kubernetes environment.


Nit: "NeMo" -> "NeMo Curator"

ryantwolf · 2024-07-19T20:37:28Z

docs/user-guide/aws-examples/eks.rst

+
+This approach is allows the execution of NeMo modules within the scheduler pod from a separate client pod. This separation ensures that the client pod can be provisioned with specific permissions tailored for executing commands and accessing resources within the Kubernetes environment.
+
+Moreover, deploying this client pod can be orchestrated by another service such as AWS Batch, facilitating scalable and efficient management of computational tasks within Kubernetes clusters.


Thank you for this explanation! This makes a lot of sense. Do you mind moving it to before you spin up the client pod so the user has background into why they would be doing this?

ryantwolf · 2024-07-19T20:38:29Z

docs/user-guide/aws-examples/eks.rst

+     storageClassName: efs-sc
+     resources:
+       requests:
+         storage: 150Gi


Not too familiar with EKS, but should this be 150GiB instead?

Referred these examples: https://github.com/kubernetes-sigs/aws-efs-csi-driver/blob/master/examples/kubernetes/dynamic_provisioning/specs/pod.yaml and it works.

Great, thanks for checking.

…client Signed-off-by: dpadmanabhan <dpadmanabhan@nvidia.com>

…client

ryantwolf

I left a few more comments on some wording choices. Also it seems like most of my feedback was not addressed, so please address it.

ryantwolf · 2024-07-30T22:12:25Z

docs/user-guide/index.rst

@@ -36,6 +36,9 @@
 :ref:`Next Steps <data-curator-next-steps>`
   Now that you've curated your data, let's discuss where to go next in the NeMo Framework to put it to good use.

+`NeMo Curator on AWS <https://github.com/NVIDIA/NeMo-Curator/tree/main/docs/user-guide/aws-examples/>`__


Can you follow the pattern of other elements in the list and do something like:

:ref:`NeMo Curator on AWS <data-curator-aws-examples>`

Then create an aws-examples/index.rst with something like this:

.. _data-curator-aws-examples: ================== AWS ================== .. toctree:: :maxdepth: 4 :titlesonly: eks.rst

ryantwolf · 2024-07-30T22:17:52Z

docs/user-guide/aws-examples/eks.rst

+
+For more details, refer to `EKS documentation <https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html>`__
+
+This guide covers all essential prerequisites. It includes an example demonstrating how to create an EFS storage class and offers step-by-step instructions for setting up an EFS Persistent Volume Claim to dynamically provisioning Kubernetes Persistent Volume. Furthermore, it outlines the required steps to deploy a Dask cluster and delves into utilizing the Kubernetes Python client library to assign NeMo-Curator tasks to the Dask scheduler.


Nit: "assign NeMo-Curator tasks to the Dask scheduler" -> "run NeMo Curator scripts"

ryantwolf · 2024-07-30T22:19:28Z

docs/user-guide/aws-examples/eks.rst

+     storageClassName: efs-sc
+     resources:
+       requests:
+         storage: 150Gi


Great, thanks for checking.

ryantwolf · 2024-07-30T22:20:22Z

docs/user-guide/aws-examples/eks.rst

+Dask cluster creation:
+----------------------
+
+Please refer index.rst for instructions on creating a Docker secret to utilize the NeMo image and upload data to the PVC created in the previous step.


Can you fix this link and also take into account my other feedback about calling out the relevant sections?

dpadmanabhan03 force-pushed the main branch from 07d0df1 to 2f5ff1d Compare July 9, 2024 17:00

ryantwolf self-requested a review July 15, 2024 16:02

ryantwolf reviewed Jul 15, 2024

View reviewed changes

dpadmanabhan03 force-pushed the main branch 13 times, most recently from 30a5e68 to d489d5e Compare July 19, 2024 19:16

ryantwolf reviewed Jul 19, 2024

View reviewed changes

Adding an example for executing NeMo modules using kubernetes Python …

909b879

…client Signed-off-by: dpadmanabhan <dpadmanabhan@nvidia.com>

dpadmanabhan03 force-pushed the main branch from d489d5e to 909b879 Compare July 23, 2024 05:32

Merge branch 'NVIDIA:main' into main

1c71305

dpadmanabhan03 force-pushed the main branch 2 times, most recently from 120ca1b to 88c44a0 Compare July 23, 2024 06:06

Adding an example for executing NeMo modules using kubernetes Python …

a134f10

…client

dpadmanabhan03 force-pushed the main branch from 88c44a0 to a134f10 Compare July 24, 2024 04:32

ryantwolf reviewed Jul 30, 2024

View reviewed changes

sarahyurick added the documentation Improvements or additions to documentation label Oct 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding an example for executing NeMo modules using kubernetes Python … #148

Adding an example for executing NeMo modules using kubernetes Python … #148

dpadmanabhan03 commented Jul 9, 2024

ryantwolf left a comment •

edited

Loading

ryantwolf left a comment

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024

ryantwolf Jul 19, 2024 •

edited

Loading

dpadmanabhan03 Jul 24, 2024

ryantwolf Jul 30, 2024

ryantwolf left a comment

ryantwolf Jul 30, 2024

ryantwolf Jul 30, 2024

ryantwolf Jul 30, 2024

ryantwolf Jul 30, 2024

		kubectl exec client-pod -- python3 kubeclient.py --command "add_id --scheduler-address localhost:8786 --input-data-dir=/nemo-workspace/arxiv --output-data-dir=/nemo-workspace/arxiv-addid/"


		This approach is allows the execution of NeMo modules within the scheduler pod from a separate client pod. This separation ensures that the client pod can be provisioned with specific permissions tailored for executing commands and accessing resources within the Kubernetes environment.


		This approach is allows the execution of NeMo modules within the scheduler pod from a separate client pod. This separation ensures that the client pod can be provisioned with specific permissions tailored for executing commands and accessing resources within the Kubernetes environment.

		Moreover, deploying this client pod can be orchestrated by another service such as AWS Batch, facilitating scalable and efficient management of computational tasks within Kubernetes clusters.


		For more details, refer to `EKS documentation <https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html>`__

		This guide covers all essential prerequisites. It includes an example demonstrating how to create an EFS storage class and offers step-by-step instructions for setting up an EFS Persistent Volume Claim to dynamically provisioning Kubernetes Persistent Volume. Furthermore, it outlines the required steps to deploy a Dask cluster and delves into utilizing the Kubernetes Python client library to assign NeMo-Curator tasks to the Dask scheduler.

Adding an example for executing NeMo modules using kubernetes Python … #148

Are you sure you want to change the base?

Adding an example for executing NeMo modules using kubernetes Python … #148

Conversation

dpadmanabhan03 commented Jul 9, 2024

Description

Usage

Checklist

ryantwolf left a comment • edited Loading

Choose a reason for hiding this comment

ryantwolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryantwolf Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryantwolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryantwolf left a comment •

edited

Loading

ryantwolf Jul 19, 2024 •

edited

Loading