Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Outdated GCP getting started guide[skip ci] #5266

Merged
merged 4 commits into from
Apr 24, 2022
Merged

Update Outdated GCP getting started guide[skip ci] #5266

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b19154e
plugin version is out of date, update the versions and remove some co…
nvliyuan Apr 18, 2022
5ed3ec1
test markdown check issue
nvliyuan Apr 18, 2022
130d5a5
revert markdown links checker changes
nvliyuan Apr 18, 2022
aaa28e0
update copyright
nvliyuan Apr 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 5 additions & 19 deletions docs/get-started/gpu_dataproc_packages_ubuntu_sample.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -126,12 +126,11 @@ function configure_gpu_isolation() {
chmod a+rwx -R /sys/fs/cgroup/devices
jlowe marked this conversation as resolved.
Show resolved Hide resolved
}

readonly RAPIDS_VERSION="0.19"
readonly DEFAULT_SPARK_RAPIDS_VERSION="0.5.0"
readonly DEFAULT_SPARK_RAPIDS_VERSION="22.02.0"
readonly DEFAULT_CUDA_VERSION="11.0"
readonly DEFAULT_CUDF_VERSION="0.19.2"
readonly DEFAULT_XGBOOST_VERSION="1.3.0"
readonly DEFAULT_XGBOOST_GPU_SUB_VERSION="0.1.0"
readonly DEFAULT_CUDF_VERSION="22.02.0"
readonly DEFAULT_XGBOOST_VERSION="1.4.2"
readonly DEFAULT_XGBOOST_GPU_SUB_VERSION="0.2.0"
readonly SPARK_VERSION="3.0"

readonly CUDF_VERSION=${DEFAULT_CUDF_VERSION}
Expand Down Expand Up @@ -168,23 +167,10 @@ function configure_spark() {
# query explain output won't show GPU operator, if user have doubt
# they can uncomment the line before to see the GPU plan explan, but AQE on give user the best performance.
# spark.sql.adaptive.enabled=false
spark.rapids.sql.concurrentGpuTasks=2
spark.executor.resource.gpu.amount=1
spark.executor.cores=4
spark.executor.memory=8G
spark.task.cpus=1
spark.task.resource.gpu.amount=0.25
spark.rapids.memory.pinnedPool.size=2G
spark.executor.memoryOverhead=2G
spark.plugins=com.nvidia.spark.SQLPlugin
spark.executor.extraJavaOptions='-Dai.rapids.cudf.prefer-pinned=true'
spark.locality.wait=0s
spark.executor.resource.gpu.discoveryScript=/usr/lib/spark/scripts/gpu/getGpusResources.sh
spark.sql.shuffle.partitions=48
spark.sql.files.maxPartitionBytes=512m
spark.submit.pyFiles=/usr/lib/spark/jars/xgboost4j-spark_${SPARK_VERSION}-${XGBOOST_VERSION}-${XGBOOST_GPU_SUB_VERSION}.jar
spark.dynamicAllocation.enabled=false
spark.shuffle.service.enabled=false
###### END : RAPIDS properties for Spark ${SPARK_VERSION} ######
EOF
}
Expand Down