Skip to content

Commit

Permalink
Fix databricks doc for limitations. (#4755)
Browse files Browse the repository at this point in the history
Signed-off-by: Hao Zhu <hazhu@nvidia.com>
  • Loading branch information
viadea authored Feb 11, 2022
1 parent 42c7eb1 commit 7586051
Showing 1 changed file with 9 additions and 9 deletions.
18 changes: 9 additions & 9 deletions docs/get-started/getting-started-databricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,12 @@ The number of GPUs per node dictates the number of Spark executors that can run
1. Adaptive query execution(AQE) and Delta optimization write do not work. These should be disabled
when using the plugin. Queries may still see significant speedups even with AQE disabled.

```bash
spark.databricks.delta.optimizeWrite.enabled false
spark.sql.adaptive.enabled false
```
```bash
spark.databricks.delta.optimizeWrite.enabled false
spark.sql.adaptive.enabled false
```

See [issue-1059](https://github.com/NVIDIA/spark-rapids/issues/1059) for more detail.
See [issue-1059](https://github.com/NVIDIA/spark-rapids/issues/1059) for more detail.

2. Dynamic partition pruning(DPP) does not work. This results in poor performance for queries which
would normally benefit from DPP. See
Expand All @@ -42,10 +42,10 @@ when using the plugin. Queries may still see significant speedups even with AQE

4. Cannot spin off multiple executors on a multi-GPU node.

Even though it is possible to set `spark.executor.resource.gpu.amount=N` (where N is the number
of GPUs per node) in the in Spark Configuration tab, Databricks overrides this to
`spark.executor.resource.gpu.amount=1`. This will result in failed executors when starting the
cluster.
Even though it is possible to set `spark.executor.resource.gpu.amount=1` in the in Spark
Configuration tab, Databricks overrides this to `spark.executor.resource.gpu.amount=N`
(where N is the number of GPUs per node). This will result in failed executors when starting the
cluster.

5. Databricks makes changes to the runtime without notification.

Expand Down

0 comments on commit 7586051

Please sign in to comment.