From f80cb6ac86e5d4e29277938e50fbfc4770c17cb9 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Fri, 5 Mar 2021 21:28:41 -0800 Subject: [PATCH 01/12] doing some test --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2b758c3665b..44ddf4f44e1 100644 --- a/README.md +++ b/README.md @@ -50,3 +50,4 @@ Currently, we are working with XGBoost to try to provide this integration out of You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding. +test From 47fdca4df7de6f93b464498c01309485fef25574 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Fri, 5 Mar 2021 22:05:15 -0800 Subject: [PATCH 02/12] Revert "doing some test" This reverts commit f80cb6ac86e5d4e29277938e50fbfc4770c17cb9. Rollback changes to README --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 44ddf4f44e1..2b758c3665b 100644 --- a/README.md +++ b/README.md @@ -50,4 +50,3 @@ Currently, we are working with XGBoost to try to provide this integration out of You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding. -test From ede56bd9aa02b0a3b67605c75c13d83d1efd6e19 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Mon, 8 Mar 2021 11:18:04 -0800 Subject: [PATCH 03/12] Update download.md Add Version Matrix in a spreadsheet. --- docs/download.md | 220 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 199 insertions(+), 21 deletions(-) diff --git a/docs/download.md b/docs/download.md index f743eddfb38..647da901b68 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,6 +4,205 @@ title: Download nav_order: 3 --- + +## Download link and version matrix + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RAPIDS + Accelerator For
+ Apache Spark1		RAPIDS
+ (cuDF)2		CUDA & + NVIDIA Driver3SparkHadoopOSPythonScalaJAVAGPU RequirementGPU Tested
0.10.1410.1.2 & 418.87+3.02.10+, + 3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.x2.128NVIDIA Pascal™ or betterV100, + T4
0.1410.2 + & 440.33+
0.20.1510.1.2 & 418.87+3.0, + 3.0.12.10+, + 3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.x2.128NVIDIA Pascal™ or betterV100, + T4,A100
0.1510.2 + & 440.33+
0.1511.0 + & 450.36+
0.30.1710.1.2 & 418.87+3.0, + 3.0.1
+ Databricks 7.3 ML LTS Runtime
+ GCP Dataproc 2.0 		2.10+, + 3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.6+2.128NVIDIA Pascal™ or betterV100, + T4,A100
0.1710.2 + & 440.33+
0.1711.0 + & 450.36+
0.40.1810.1.2 & 418.87+3.0, + 3.0.1, 3.0.2, 3.1.1
+ Databricks 7.3 ML LTS Runtime
+ GCP Dataproc 2.0
+ AWS EMR 6.3.0 (?)		2.10+ + ,3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.6+2.128NVIDIA Pascal™ or betterV100, + T4,A100
0.1810.2 + & 440.33+
0.1811.0 + & 450.36+
+ +* Note 1: [RAPIDS Accelerator For Apache Spark](https://github.com/NVIDIA/spark-rapids) provides a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. + +* Note 2: [RAPIDS cuDF](https://github.com/rapidsai/cudf) project is the cornerstone for "RAPIDS Accelerator For Apache Spark". + +* Note 3: CUDA and NVIDIA Driver need to be installed on the machines where CUDA-capable GPU is attached. + +* Note 4: Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. + ## Release v0.4.0 New functionality for the release includes @@ -45,11 +244,6 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 -### Download v0.4.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) ## Release v0.3.0 This release includes additional performance improvements, including @@ -88,12 +282,6 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 -### Download v0.3.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) - ## Release v0.2.0 This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was @@ -134,11 +322,6 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 -### Download v0.2.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) ## Release v0.1.0 @@ -160,10 +343,5 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 -### Download v0.1.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) - From 744bb42da3f5f5042606a310cc0540e7ac89e18b Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Mon, 8 Mar 2021 11:18:04 -0800 Subject: [PATCH 04/12] Update download.md Add Version Matrix in a spreadsheet. Signed-off-by: Hao Zhu --- docs/download.md | 220 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 199 insertions(+), 21 deletions(-) diff --git a/docs/download.md b/docs/download.md index f743eddfb38..647da901b68 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,6 +4,205 @@ title: Download nav_order: 3 --- + +## Download link and version matrix + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
RAPIDS + Accelerator For
+ Apache Spark1		RAPIDS
+ (cuDF)2		CUDA & + NVIDIA Driver3SparkHadoopOSPythonScalaJAVAGPU RequirementGPU Tested
0.10.1410.1.2 & 418.87+3.02.10+, + 3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.x2.128NVIDIA Pascal™ or betterV100, + T4
0.1410.2 + & 440.33+
0.20.1510.1.2 & 418.87+3.0, + 3.0.12.10+, + 3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.x2.128NVIDIA Pascal™ or betterV100, + T4,A100
0.1510.2 + & 440.33+
0.1511.0 + & 450.36+
0.30.1710.1.2 & 418.87+3.0, + 3.0.1
+ Databricks 7.3 ML LTS Runtime
+ GCP Dataproc 2.0 		2.10+, + 3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.6+2.128NVIDIA Pascal™ or betterV100, + T4,A100
0.1710.2 + & 440.33+
0.1711.0 + & 450.36+
0.40.1810.1.2 & 418.87+3.0, + 3.0.1, 3.0.2, 3.1.1
+ Databricks 7.3 ML LTS Runtime
+ GCP Dataproc 2.0
+ AWS EMR 6.3.0 (?)		2.10+ + ,3.1.1+
+ 3.1.1 for nvidia-docker version 2		Ubuntu + 16.04
+ Ubuntu 18.04
+ CentOS 7		3.6+2.128NVIDIA Pascal™ or betterV100, + T4,A100
0.1810.2 + & 440.33+
0.1811.0 + & 450.36+
+ +* Note 1: [RAPIDS Accelerator For Apache Spark](https://github.com/NVIDIA/spark-rapids) provides a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. + +* Note 2: [RAPIDS cuDF](https://github.com/rapidsai/cudf) project is the cornerstone for "RAPIDS Accelerator For Apache Spark". + +* Note 3: CUDA and NVIDIA Driver need to be installed on the machines where CUDA-capable GPU is attached. + +* Note 4: Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. + ## Release v0.4.0 New functionality for the release includes @@ -45,11 +244,6 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 -### Download v0.4.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) ## Release v0.3.0 This release includes additional performance improvements, including @@ -88,12 +282,6 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 -### Download v0.3.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) - ## Release v0.2.0 This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was @@ -134,11 +322,6 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 -### Download v0.2.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) ## Release v0.1.0 @@ -160,10 +343,5 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 -### Download v0.1.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) - From 4a82093b2c885ff23aefbd04b8768fc0b66b4449 Mon Sep 17 00:00:00 2001 From: Hao Zhu Date: Mon, 8 Mar 2021 12:29:57 -0800 Subject: [PATCH 05/12] Revert "Update download.md" This reverts commit 744bb42da3f5f5042606a310cc0540e7ac89e18b. --- docs/download.md | 220 +++++------------------------------------------ 1 file changed, 21 insertions(+), 199 deletions(-) diff --git a/docs/download.md b/docs/download.md index 647da901b68..f743eddfb38 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,205 +4,6 @@ title: Download nav_order: 3 --- - -## Download link and version matrix - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RAPIDS - Accelerator For
- Apache Spark1		RAPIDS
- (cuDF)2		CUDA & - NVIDIA Driver3SparkHadoopOSPythonScalaJAVAGPU RequirementGPU Tested
0.10.1410.1.2 & 418.87+3.02.10+, - 3.1.1+
- 3.1.1 for nvidia-docker version 2		Ubuntu - 16.04
- Ubuntu 18.04
- CentOS 7		3.x2.128NVIDIA Pascal™ or betterV100, - T4
0.1410.2 - & 440.33+
0.20.1510.1.2 & 418.87+3.0, - 3.0.12.10+, - 3.1.1+
- 3.1.1 for nvidia-docker version 2		Ubuntu - 16.04
- Ubuntu 18.04
- CentOS 7		3.x2.128NVIDIA Pascal™ or betterV100, - T4,A100
0.1510.2 - & 440.33+
0.1511.0 - & 450.36+
0.30.1710.1.2 & 418.87+3.0, - 3.0.1
- Databricks 7.3 ML LTS Runtime
- GCP Dataproc 2.0 		2.10+, - 3.1.1+
- 3.1.1 for nvidia-docker version 2		Ubuntu - 16.04
- Ubuntu 18.04
- CentOS 7		3.6+2.128NVIDIA Pascal™ or betterV100, - T4,A100
0.1710.2 - & 440.33+
0.1711.0 - & 450.36+
0.40.1810.1.2 & 418.87+3.0, - 3.0.1, 3.0.2, 3.1.1
- Databricks 7.3 ML LTS Runtime
- GCP Dataproc 2.0
- AWS EMR 6.3.0 (?)		2.10+ - ,3.1.1+
- 3.1.1 for nvidia-docker version 2		Ubuntu - 16.04
- Ubuntu 18.04
- CentOS 7		3.6+2.128NVIDIA Pascal™ or betterV100, - T4,A100
0.1810.2 - & 440.33+
0.1811.0 - & 450.36+
- -* Note 1: [RAPIDS Accelerator For Apache Spark](https://github.com/NVIDIA/spark-rapids) provides a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. - -* Note 2: [RAPIDS cuDF](https://github.com/rapidsai/cudf) project is the cornerstone for "RAPIDS Accelerator For Apache Spark". - -* Note 3: CUDA and NVIDIA Driver need to be installed on the machines where CUDA-capable GPU is attached. - -* Note 4: Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. - ## Release v0.4.0 New functionality for the release includes @@ -244,6 +45,11 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 +### Download v0.4.0 +* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) +* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) +* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) +* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) ## Release v0.3.0 This release includes additional performance improvements, including @@ -282,6 +88,12 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 +### Download v0.3.0 +* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) +* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) +* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) +* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) + ## Release v0.2.0 This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was @@ -322,6 +134,11 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 +### Download v0.2.0 +* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) +* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) +* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) +* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) ## Release v0.1.0 @@ -343,5 +160,10 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 +### Download v0.1.0 +* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) +* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) +* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) + From d1ae8a5d32b13b82d3b3b3b6c5cb3529e3a01e82 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 15:09:05 -0800 Subject: [PATCH 06/12] Change the download link format Added several words to explain the relationship between RAPIDS Accelerator For Apache Spark, RAPIDS cuDF and CUDA&NVIDIA Driver. Changed the download link format. --- docs/download.md | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/docs/download.md b/docs/download.md index f743eddfb38..3d260001faf 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,6 +4,14 @@ title: Download nav_order: 3 --- +[RAPIDS Accelerator For Apache Spark provides](https://github.com/NVIDIA/spark-rapids) a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. + +[RAPIDS cuDF](https://github.com/rapidsai/cudf) project is the cornerstone for [RAPIDS Accelerator For Apache Spark provides](https://github.com/NVIDIA/spark-rapids). + +CUDA and NVIDIA Driver need to be installed on the machine where CUDA-capable GPU is attached. + +Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. + ## Release v0.4.0 New functionality for the release includes @@ -46,10 +54,11 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 ### Download v0.4.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) +* Download [RAPIDS Accelerator For Apache Spark v0.4.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) +* Download RAPIDS cuDF 0.18 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) ## Release v0.3.0 This release includes additional performance improvements, including @@ -89,11 +98,12 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 ### Download v0.3.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) - +* Download [RAPIDS Accelerator For Apache Spark v0.3.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) +* Download RAPIDS cuDF 0.17 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) + ## Release v0.2.0 This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was @@ -135,10 +145,12 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 ### Download v0.2.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) +* Download [RAPIDS Accelerator For Apache Spark v0.2.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) +* Download RAPIDS cuDF 0.15 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) + ## Release v0.1.0 @@ -161,9 +173,9 @@ Software Requirements: ### Download v0.1.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) - +* Download [RAPIDS Accelerator For Apache Spark v0.1.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) +* Download RAPIDS cuDF 0.14 for your system: + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) From 1580387265afb091d527c41e55c8bfc6b4e864ac Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 15:09:05 -0800 Subject: [PATCH 07/12] Change the download link format Added several words to explain the relationship between RAPIDS Accelerator For Apache Spark, RAPIDS cuDF and CUDA&NVIDIA Driver. Changed the download link format. Signed-off-by: Hao Zhu <9665750+viadea@users.noreply.github.com> --- docs/download.md | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/docs/download.md b/docs/download.md index f743eddfb38..3d260001faf 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,6 +4,14 @@ title: Download nav_order: 3 --- +[RAPIDS Accelerator For Apache Spark provides](https://github.com/NVIDIA/spark-rapids) a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. + +[RAPIDS cuDF](https://github.com/rapidsai/cudf) project is the cornerstone for [RAPIDS Accelerator For Apache Spark provides](https://github.com/NVIDIA/spark-rapids). + +CUDA and NVIDIA Driver need to be installed on the machine where CUDA-capable GPU is attached. + +Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. + ## Release v0.4.0 New functionality for the release includes @@ -46,10 +54,11 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 ### Download v0.4.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) +* Download [RAPIDS Accelerator For Apache Spark v0.4.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) +* Download RAPIDS cuDF 0.18 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) ## Release v0.3.0 This release includes additional performance improvements, including @@ -89,11 +98,12 @@ Software Requirements: Python 3.6+, Scala 2.12, Java 8 ### Download v0.3.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) - +* Download [RAPIDS Accelerator For Apache Spark v0.3.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) +* Download RAPIDS cuDF 0.17 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) + ## Release v0.2.0 This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was @@ -135,10 +145,12 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 ### Download v0.2.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) -* [cuDF 11.0 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) +* Download [RAPIDS Accelerator For Apache Spark v0.2.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) +* Download RAPIDS cuDF 0.15 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) + ## Release v0.1.0 @@ -161,9 +173,9 @@ Software Requirements: ### Download v0.1.0 -* [RAPIDS Spark Package](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) -* [cuDF 10.2 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) -* [cuDF 10.1 Package](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) - +* Download [RAPIDS Accelerator For Apache Spark v0.1.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) +* Download RAPIDS cuDF 0.14 for your system: + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) From 3f467e12e4fbe4cc01ccd00ea2dbc57c8c52752e Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 15:20:18 -0800 Subject: [PATCH 08/12] Reorder the download links, requirements and release notes Reorder the download links, requirements and release notes --- docs/download.md | 109 ++++++++++++++++++++++++++++------------------- 1 file changed, 65 insertions(+), 44 deletions(-) diff --git a/docs/download.md b/docs/download.md index 3d260001faf..943711716bb 100644 --- a/docs/download.md +++ b/docs/download.md @@ -13,7 +13,31 @@ CUDA and NVIDIA Driver need to be installed on the machine where CUDA-capable GP Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. ## Release v0.4.0 +### Download v0.4.0 +* Download [RAPIDS Accelerator For Apache Spark v0.4.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) +* Download RAPIDS cuDF 0.18 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) + +### Requirements +Hardware Requirements: + + GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) + +Software Requirements: + + OS: Ubuntu 16.04, Ubuntu 18.04 or CentOS 7 + + CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ + + Apache Spark 3.0, 3.0.1, 3.0.2, 3.1.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 + + Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) + + Python 3.6+, Scala 2.12, Java 8 +### Release Notes New functionality for the release includes * Decimal support up to 64 bit, including reading and writing decimal from Parquet (can be enabled by setting `spark.rapids.sql.decimalType.enabled` to True) @@ -37,6 +61,20 @@ The list of all supported operations is provided [here](supported_ops.md). For a detailed list of changes, please refer to the [CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md). + + + + + +## Release v0.3.0 +### Download v0.3.0 +* Download [RAPIDS Accelerator For Apache Spark v0.3.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) +* Download RAPIDS cuDF 0.17 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) + +### Requirements Hardware Requirements: GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) @@ -47,20 +85,13 @@ Software Requirements: CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ - Apache Spark 3.0, 3.0.1, 3.0.2, 3.1.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 + Apache Spark 3.0, 3.0.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) Python 3.6+, Scala 2.12, Java 8 -### Download v0.4.0 -* Download [RAPIDS Accelerator For Apache Spark v0.4.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) -* Download RAPIDS cuDF 0.18 for your system: - * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) - -## Release v0.3.0 +### Release Notes This release includes additional performance improvements, including * Use of per thread default stream to make more efficient use of the GPU * Further supporting Spark's adaptive query execution, with more rewritten query plans now able to @@ -81,6 +112,20 @@ The list of all supported operations is provided [here](supported_ops.md). For a detailed list of changes, please refer to the [CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md). + + + + + +## Release v0.2.0 +### Download v0.2.0 +* Download [RAPIDS Accelerator For Apache Spark v0.2.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) +* Download RAPIDS cuDF 0.15 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) + +### Requirements Hardware Requirements: GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) @@ -91,20 +136,13 @@ Software Requirements: CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ - Apache Spark 3.0, 3.0.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 + Apache Spark 3.0, 3.0.1 Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) - Python 3.6+, Scala 2.12, Java 8 - -### Download v0.3.0 -* Download [RAPIDS Accelerator For Apache Spark v0.3.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) -* Download RAPIDS cuDF 0.17 for your system: - * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) - -## Release v0.2.0 + Python 3.x, Scala 2.12, Java 8 + +### Release Notes This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was included in Spark 3.0 that alters the physical execution plan dynamically to improve the performance @@ -128,32 +166,19 @@ The list of all supported operations is provided For a detailed list of changes, please refer to the [CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md). -Hardware Requirements: - GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) - -Software Requirements: - OS: Ubuntu 16.04, Ubuntu 18.04 or CentOS 7 - - CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ - - Apache Spark 3.0, 3.0.1 - - Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) - - Python 3.x, Scala 2.12, Java 8 -### Download v0.2.0 -* Download [RAPIDS Accelerator For Apache Spark v0.2.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) -* Download RAPIDS cuDF 0.15 for your system: - * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) ## Release v0.1.0 +### Download v0.1.0 +* Download [RAPIDS Accelerator For Apache Spark v0.1.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) +* Download RAPIDS cuDF 0.14 for your system: + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) +### Requirements Hardware Requirements: GPU Architecture: NVIDIA Pascal™ or better (Tested on V100 and T4 GPU) @@ -172,10 +197,6 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 -### Download v0.1.0 -* Download [RAPIDS Accelerator For Apache Spark v0.1.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) -* Download RAPIDS cuDF 0.14 for your system: - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) + From ede9d97f93a59ac68e96c612f095189187d13255 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 15:20:18 -0800 Subject: [PATCH 09/12] Reorder the download links, requirements and release notes Reorder the download links, requirements and release notes Signed-off-by: Hao Zhu <9665750+viadea@users.noreply.github.com> --- docs/download.md | 109 ++++++++++++++++++++++++++++------------------- 1 file changed, 65 insertions(+), 44 deletions(-) diff --git a/docs/download.md b/docs/download.md index 3d260001faf..943711716bb 100644 --- a/docs/download.md +++ b/docs/download.md @@ -13,7 +13,31 @@ CUDA and NVIDIA Driver need to be installed on the machine where CUDA-capable GP Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. ## Release v0.4.0 +### Download v0.4.0 +* Download [RAPIDS Accelerator For Apache Spark v0.4.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) +* Download RAPIDS cuDF 0.18 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) + +### Requirements +Hardware Requirements: + + GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) + +Software Requirements: + + OS: Ubuntu 16.04, Ubuntu 18.04 or CentOS 7 + + CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ + + Apache Spark 3.0, 3.0.1, 3.0.2, 3.1.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 + + Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) + + Python 3.6+, Scala 2.12, Java 8 +### Release Notes New functionality for the release includes * Decimal support up to 64 bit, including reading and writing decimal from Parquet (can be enabled by setting `spark.rapids.sql.decimalType.enabled` to True) @@ -37,6 +61,20 @@ The list of all supported operations is provided [here](supported_ops.md). For a detailed list of changes, please refer to the [CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md). + + + + + +## Release v0.3.0 +### Download v0.3.0 +* Download [RAPIDS Accelerator For Apache Spark v0.3.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) +* Download RAPIDS cuDF 0.17 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) + +### Requirements Hardware Requirements: GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) @@ -47,20 +85,13 @@ Software Requirements: CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ - Apache Spark 3.0, 3.0.1, 3.0.2, 3.1.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 + Apache Spark 3.0, 3.0.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) Python 3.6+, Scala 2.12, Java 8 -### Download v0.4.0 -* Download [RAPIDS Accelerator For Apache Spark v0.4.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.4.0/rapids-4-spark_2.12-0.4.0.jar) -* Download RAPIDS cuDF 0.18 for your system: - * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda11.jar) - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.18/cudf-0.18-cuda10-1.jar) - -## Release v0.3.0 +### Release Notes This release includes additional performance improvements, including * Use of per thread default stream to make more efficient use of the GPU * Further supporting Spark's adaptive query execution, with more rewritten query plans now able to @@ -81,6 +112,20 @@ The list of all supported operations is provided [here](supported_ops.md). For a detailed list of changes, please refer to the [CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md). + + + + + +## Release v0.2.0 +### Download v0.2.0 +* Download [RAPIDS Accelerator For Apache Spark v0.2.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) +* Download RAPIDS cuDF 0.15 for your system: + * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) + +### Requirements Hardware Requirements: GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) @@ -91,20 +136,13 @@ Software Requirements: CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ - Apache Spark 3.0, 3.0.1, Databricks 7.3 ML LTS Runtime, or GCP Dataproc 2.0 + Apache Spark 3.0, 3.0.1 Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) - Python 3.6+, Scala 2.12, Java 8 - -### Download v0.3.0 -* Download [RAPIDS Accelerator For Apache Spark v0.3.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.3.0/rapids-4-spark_2.12-0.3.0.jar) -* Download RAPIDS cuDF 0.17 for your system: - * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda11.jar) - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.17/cudf-0.17-cuda10-1.jar) - -## Release v0.2.0 + Python 3.x, Scala 2.12, Java 8 + +### Release Notes This is the second release of the RAPIDS Accelerator for Apache Spark. Adaptive Query Execution [SPARK-31412](https://issues.apache.org/jira/browse/SPARK-31412) is a new enhancement that was included in Spark 3.0 that alters the physical execution plan dynamically to improve the performance @@ -128,32 +166,19 @@ The list of all supported operations is provided For a detailed list of changes, please refer to the [CHANGELOG](https://github.com/NVIDIA/spark-rapids/blob/main/CHANGELOG.md). -Hardware Requirements: - GPU Architecture: NVIDIA Pascal™ or better (Tested on V100, T4 and A100 GPU) - -Software Requirements: - OS: Ubuntu 16.04, Ubuntu 18.04 or CentOS 7 - - CUDA & Nvidia Drivers: 10.1.2 & v418.87+, 10.2 & v440.33+ or 11.0 & v450.36+ - - Apache Spark 3.0, 3.0.1 - - Apache Hadoop 2.10+ or 3.1.1+ (3.1.1 for nvidia-docker version 2) - - Python 3.x, Scala 2.12, Java 8 -### Download v0.2.0 -* Download [RAPIDS Accelerator For Apache Spark v0.2.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.2.0/rapids-4-spark_2.12-0.2.0.jar) -* Download RAPIDS cuDF 0.15 for your system: - * [For CUDA 11.0 & NVIDIA driver 450.36+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda11.jar) - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.15/cudf-0.15-cuda10-1.jar) ## Release v0.1.0 +### Download v0.1.0 +* Download [RAPIDS Accelerator For Apache Spark v0.1.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) +* Download RAPIDS cuDF 0.14 for your system: + * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) + * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) +### Requirements Hardware Requirements: GPU Architecture: NVIDIA Pascal™ or better (Tested on V100 and T4 GPU) @@ -172,10 +197,6 @@ Software Requirements: Python 3.x, Scala 2.12, Java 8 -### Download v0.1.0 -* Download [RAPIDS Accelerator For Apache Spark v0.1.0](https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/0.1.0/rapids-4-spark_2.12-0.1.0.jar) -* Download RAPIDS cuDF 0.14 for your system: - * [For CUDA 10.2 & NVIDIA driver 440.33+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-2.jar) - * [For CUDA 10.1 & NVIDIA driver 418.87+](https://repo1.maven.org/maven2/ai/rapids/cudf/0.14/cudf-0.14-cuda10-1.jar) + From 41602cb007bd1a4015ec1cd772082a2919b95071 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 15:58:49 -0800 Subject: [PATCH 10/12] Modified some introduction messages. Modified some introduction messages. --- docs/download.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/download.md b/docs/download.md index 943711716bb..c557cbf46cf 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,13 +4,13 @@ title: Download nav_order: 3 --- -[RAPIDS Accelerator For Apache Spark provides](https://github.com/NVIDIA/spark-rapids) a set of plugins for Apache Spark that leverage GPUs to accelerate processing via the RAPIDS libraries and UCX. +[RAPIDS Accelerator For Apache Spark](https://github.com/NVIDIA/spark-rapids) provides a set of plugins for Apache Spark that leverage GPUs to accelerate Dataframe and SQL processing. The accelerator is built upon the [RAPIDS cuDF project](https://github.com/rapidsai/cudf) and [UCX](https://github.com/openucx/ucx/). -[RAPIDS cuDF](https://github.com/rapidsai/cudf) project is the cornerstone for [RAPIDS Accelerator For Apache Spark provides](https://github.com/NVIDIA/spark-rapids). +The RAPIDS Accelerator For Apache Spark requires each worker node in the cluster to have [CUDA](https://developer.nvidia.com/cuda-toolkit) installed. -CUDA and NVIDIA Driver need to be installed on the machine where CUDA-capable GPU is attached. +The RAPIDS Accelerator For Apache Spark consists of two jars, a plugin jar along with the RAPIDS cuDF jar, that is either preinstalled in the Spark classpath on all nodes or submitted with each job that uses the RAPIDS Accelerator For Apache Spark. -Spark job needs 2 jars -- "RAPIDS Accelerator For Apache Spark" jar, and "RAPIDS cuDF" jar based on corresponding CUDA&NVIDIA Driver version. +Here is the [getting-started guide](https://nvidia.github.io/spark-rapids/Getting-Started/). ## Release v0.4.0 ### Download v0.4.0 From d899dda36fafcd50e36b3071e40f0ad174f12fc9 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 16:16:19 -0800 Subject: [PATCH 11/12] Rewording in download page. Reword based on suggestions. --- docs/download.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/download.md b/docs/download.md index c557cbf46cf..349a3ec567d 100644 --- a/docs/download.md +++ b/docs/download.md @@ -4,13 +4,13 @@ title: Download nav_order: 3 --- -[RAPIDS Accelerator For Apache Spark](https://github.com/NVIDIA/spark-rapids) provides a set of plugins for Apache Spark that leverage GPUs to accelerate Dataframe and SQL processing. The accelerator is built upon the [RAPIDS cuDF project](https://github.com/rapidsai/cudf) and [UCX](https://github.com/openucx/ucx/). +[RAPIDS Accelerator For Apache Spark](https://github.com/NVIDIA/spark-rapids) provides a set of plugins for Apache Spark that leverage GPUs to accelerate Dataframe and SQL processing. -The RAPIDS Accelerator For Apache Spark requires each worker node in the cluster to have [CUDA](https://developer.nvidia.com/cuda-toolkit) installed. +The accelerator is built upon the [RAPIDS cuDF project](https://github.com/rapidsai/cudf) and [UCX](https://github.com/openucx/ucx/). -The RAPIDS Accelerator For Apache Spark consists of two jars, a plugin jar along with the RAPIDS cuDF jar, that is either preinstalled in the Spark classpath on all nodes or submitted with each job that uses the RAPIDS Accelerator For Apache Spark. +The RAPIDS Accelerator For Apache Spark requires each worker node in the cluster to have [CUDA](https://developer.nvidia.com/cuda-toolkit) installed. -Here is the [getting-started guide](https://nvidia.github.io/spark-rapids/Getting-Started/). +The RAPIDS Accelerator For Apache Spark consists of two jars, a plugin jar along with the RAPIDS cuDF jar, that is either preinstalled in the Spark classpath on all nodes or submitted with each job that uses the RAPIDS Accelerator For Apache Spark. See the [getting-started guide](https://nvidia.github.io/spark-rapids/Getting-Started/) for more details. ## Release v0.4.0 ### Download v0.4.0 From a069a4ca48d0dc1f5b7064edafb9a71419de2709 Mon Sep 17 00:00:00 2001 From: Hao Zhu <9665750+viadea@users.noreply.github.com> Date: Mon, 8 Mar 2021 17:36:20 -0800 Subject: [PATCH 12/12] word wrapping Did some word wrapping work based on suggestion. --- docs/download.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/download.md b/docs/download.md index 349a3ec567d..a0b55f76aa2 100644 --- a/docs/download.md +++ b/docs/download.md @@ -10,7 +10,9 @@ The accelerator is built upon the [RAPIDS cuDF project](https://github.com/rapid The RAPIDS Accelerator For Apache Spark requires each worker node in the cluster to have [CUDA](https://developer.nvidia.com/cuda-toolkit) installed. -The RAPIDS Accelerator For Apache Spark consists of two jars, a plugin jar along with the RAPIDS cuDF jar, that is either preinstalled in the Spark classpath on all nodes or submitted with each job that uses the RAPIDS Accelerator For Apache Spark. See the [getting-started guide](https://nvidia.github.io/spark-rapids/Getting-Started/) for more details. +The RAPIDS Accelerator For Apache Spark consists of two jars: a plugin jar along with the RAPIDS cuDF jar, +that is either preinstalled in the Spark classpath on all nodes or submitted with each job that uses the +RAPIDS Accelerator For Apache Spark. See the [getting-started guide](https://nvidia.github.io/spark-rapids/Getting-Started/) for more details. ## Release v0.4.0 ### Download v0.4.0