Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix databricks doc for limitations.[skip ci] #4755

Merged
merged 1 commit into from
Feb 11, 2022
Merged

Fix databricks doc for limitations.[skip ci] #4755

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/get-started/getting-started-databricks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@ The number of GPUs per node dictates the number of Spark executors that can run
1. Adaptive query execution(AQE) and Delta optimization write do not work. These should be disabled
when using the plugin. Queries may still see significant speedups even with AQE disabled.

```bash
spark.databricks.delta.optimizeWrite.enabled false
spark.sql.adaptive.enabled false
```
```bash
spark.databricks.delta.optimizeWrite.enabled false
spark.sql.adaptive.enabled false
```

See [issue-1059](https://github.com/NVIDIA/spark-rapids/issues/1059) for more detail.
See [issue-1059](https://github.com/NVIDIA/spark-rapids/issues/1059) for more detail.

2. Dynamic partition pruning(DPP) does not work. This results in poor performance for queries which
would normally benefit from DPP. See
Expand All @@ -42,10 +42,10 @@ when using the plugin. Queries may still see significant speedups even with AQE

4. Cannot spin off multiple executors on a multi-GPU node.

Even though it is possible to set `spark.executor.resource.gpu.amount=N` (where N is the number
of GPUs per node) in the in Spark Configuration tab, Databricks overrides this to
`spark.executor.resource.gpu.amount=1`. This will result in failed executors when starting the
cluster.
Even though it is possible to set `spark.executor.resource.gpu.amount=1` in the in Spark
Configuration tab, Databricks overrides this to `spark.executor.resource.gpu.amount=N`
(where N is the number of GPUs per node). This will result in failed executors when starting the
cluster.

5. Databricks makes changes to the runtime without notification.

Expand Down