Skip to content

Commit

Permalink
Modify the default value of spark.rapids.sql.explain as NOT_ON_GPU
Browse files Browse the repository at this point in the history
Signed-off-by: sinkinben <sinkinben@outlook.com>
  • Loading branch information
sinkinben committed Jun 16, 2022
1 parent 04991c6 commit bb50fd8
Show file tree
Hide file tree
Showing 4 changed files with 7 additions and 6 deletions.
7 changes: 4 additions & 3 deletions docs/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,9 +147,10 @@ An Apache Spark plan is transformed and optimized into a set of operators called
This plan is then run through a set of rules to translate it to a version that runs on the GPU.
If you want to know what will run on the GPU and what will not along with an explanation why you
can set [spark.rapids.sql.explain](configs.md#sql.explain) to `ALL`. If you just want to see the
operators not on the GPU you may set it to `NOT_ON_GPU`. Be aware that some queries end up being
broken down into multiple jobs, and in those cases a separate log message might be output for each
job. These are logged each time a query is compiled into an `RDD`, not just when the job runs.
operators not on the GPU you may set it to `NOT_ON_GPU` (which is the default setting value). Be
aware that some queries end up being broken down into multiple jobs, and in those cases a separate
log message might be output for each job. These are logged each time a query is compiled into an
`RDD`, not just when the job runs.
Because of this calling `explain` on a DataFrame will also trigger this to be logged.

The format of each line follows the pattern
Expand Down
2 changes: 1 addition & 1 deletion docs/configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Name | Description | Default Value
<a name="sql.csv.read.float.enabled"></a>spark.rapids.sql.csv.read.float.enabled|CSV reading is not 100% compatible when reading floats.|true
<a name="sql.decimalOverflowGuarantees"></a>spark.rapids.sql.decimalOverflowGuarantees|FOR TESTING ONLY. DO NOT USE IN PRODUCTION. Please see the decimal section of the compatibility documents for more information on this config.|true
<a name="sql.enabled"></a>spark.rapids.sql.enabled|Enable (true) or disable (false) sql operations on the GPU|true
<a name="sql.explain"></a>spark.rapids.sql.explain|Explain why some parts of a query were not placed on a GPU or not. Possible values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of a query that did not go on the GPU|NONE
<a name="sql.explain"></a>spark.rapids.sql.explain|Explain why some parts of a query were not placed on a GPU or not. Possible values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of a query that did not go on the GPU|NOT_ON_GPU
<a name="sql.fast.sample"></a>spark.rapids.sql.fast.sample|Option to turn on fast sample. If enable it is inconsistent with CPU sample because of GPU sample algorithm is inconsistent with CPU.|false
<a name="sql.format.avro.enabled"></a>spark.rapids.sql.format.avro.enabled|When set to true enables all avro input and output acceleration. (only input is currently supported anyways)|false
<a name="sql.format.avro.multiThreadedRead.maxNumFilesParallel"></a>spark.rapids.sql.format.avro.multiThreadedRead.maxNumFilesParallel|A limit on the maximum number of files per task processed in parallel on the CPU side before the file is sent to the GPU. This affects the amount of host memory used when reading the files in parallel. Used with MULTITHREADED reader, see spark.rapids.sql.format.avro.reader.type|2147483647
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1276,7 +1276,7 @@ object RapidsConf {
"values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of " +
"a query that did not go on the GPU")
.stringConf
.createWithDefault("NONE")
.createWithDefault("NOT_ON_GPU")

val SHIMS_PROVIDER_OVERRIDE = conf("spark.rapids.shims-provider-override")
.internal()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1279,7 +1279,7 @@ object RapidsConf {
"values are ALL: print everything, NONE: print nothing, NOT_ON_GPU: print only parts of " +
"a query that did not go on the GPU")
.stringConf
.createWithDefault("NONE")
.createWithDefault("NOT_ON_GPU")

val SHIMS_PROVIDER_OVERRIDE = conf("spark.rapids.shims-provider-override")
.internal()
Expand Down

0 comments on commit bb50fd8

Please sign in to comment.