Skip to content
This repository has been archived by the owner on Sep 18, 2023. It is now read-only.

Commit

Permalink
use arrow-unsafe allocator by default
Browse files Browse the repository at this point in the history
This patch makes arrow-unsafe allocator as default profile

arrow-netty will instance netty memory pool to reduce the
alloc/dealloc overhead which introduced big memory footprint.

local perf tests showed the performace is not heavily impacted
after switch to arrow-unsafe

one more benefit is we could remove the -XX:MaxDirectMemorySize flag
which also impacts the vanilla Spark memory usage

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
  • Loading branch information
zhouyuan committed Jun 14, 2021
1 parent 862cd21 commit 7692a08
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 3 deletions.
1 change: 0 additions & 1 deletion docs/Configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ You can add these configuration into spark-defaults.conf to enable or disable th
| spark.executorEnv.CC | To set up the location of gcc | /path/to/gcc/ |
| spark.executor.memory| To set up how much memory to be used for Spark Executor. | |
| spark.memory.offHeap.size| To set up how much memory to be used for Java OffHeap.<br /> Please notice Native SQL Engine will leverage this setting to allocate memory space for native usage even offHeap is disabled. <br /> The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Native SQL Engine | 30G |
| spark.executor.extraJavaOptions | To set up how much Direct Memory to be used for Native SQL Engine. The value is based on your system and it is recommended to set it larger if you are facing Out of Memory issue in Native SQL Engine | -XX:MaxDirectMemorySize=30G |
| spark.sql.sources.useV1SourceList | Choose to use V1 source | avro |
| spark.sql.join.preferSortMergeJoin | To turn off preferSortMergeJoin in Spark | false |
| spark.sql.extensions | To turn on Native SQL Engine Plugin | com.intel.oap.ColumnarPlugin |
Expand Down
4 changes: 2 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@
<profile>
<id>arrow-netty</id>
<activation>
<activeByDefault>true</activeByDefault>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<arrow-memory.artifact>arrow-memory-netty</arrow-memory.artifact>
Expand All @@ -222,7 +222,7 @@
<profile>
<id>arrow-unsafe</id>
<activation>
<activeByDefault>false</activeByDefault>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<arrow-memory.artifact>arrow-memory-unsafe</arrow-memory.artifact>
Expand Down

0 comments on commit 7692a08

Please sign in to comment.