Add Alluxio auto mount feature #5925

GaryShen2008 · 2022-06-28T06:40:52Z

Mount the cloud bucket to Alluxio when driver converts FileSourceScanExec to GPU plan
The Alluxio master should be the same node as Spark driver node when using this feature
Introduce new configs:
spark.rapids.alluxio.automount.enabled
spark.rapids.alluxio.bucket.regex
spark.rapids.alluxio.cmd

Signed-off-by: Gary Shen gashen@nvidia.com

Close #5872
Close #5890

GaryShen2008 · 2022-06-30T09:30:23Z

This change is to simplify the usage of Alluxio with Spark-Rapids when the Alluxio master node is the same node which runs Spark Driver app. E.g. When you run the notebook on Databricks, the spark driver will be ran on the master node and the Alluxio master is also installed on the master node.
The previous design for Alluxio usage, it requires users to run alluxio mount manually and provides a list of replacement rules like s3://bucket-foo/file -> alluxio://0.1.2.3:19998/bucket-foo/file. It's not easy to use.
This change introduces 3 new configs:
spark.rapids.alluxio.automount.enabled - enable the Alluxio auto mount feature
spark.rapids.alluxio.bucket.regex - the regex to decide which bucket to be mounted to Alluxio. Default value to match all the buckets starting with s3:// or s3a://. (Optional)
spark.rapids.alluxio.cmd - the alluxio command for a customized installation of Alluxio (Optional)

Prerequisites:
Alluxio master node is the same node which runs Spark Driver app. (So, it can call the command "alluxio fs mount" to mount buckets).

For a quick way to use this feature, you just need to set spark.rapids.alluxio.automount.enabled=true (and don't set spark.rapids.alluxio.pathsToReplace).
The workflow looks as below.
Normally, you'll set spark.hadoop.fs.s3a.access.key and spark.hadoop.fs.s3a.secret.key in spark config to read dataset on AWS S3. When you write df = spark.read.parquet("s3a://some-bucket/files"), we'll check the path of "s3a://some-bucket/files" with the spark.rapids.alluxio.bucket.regex. if it matches, we'll try to mount the bucket "some-bucket" to Alluxio's "/some-bucket" by calling a command line "alluxio fs mount --readonly --option s3a.accessKeyId=*** --option s3a.secretKey=*** /some-bucket s3a://some-bucket".
We'll read the ALLUXIO_HOME environment variable to figure out where Alluxio installed and find out the Alluxio's master IP and port from ALLUXIO_HOME/conf/alluxio-site.properties.
We'll find the access key and secret key from spark config.
Finally, we'll replace the path in Spark plans from "s3a://some-bucket/files" to "alluxio://master_ip:port/some-bucket/files" then the spark job will read data through Alluxio to cache the data to local.

GaryShen2008 · 2022-07-02T11:48:43Z

Forgot to consider the case when driver context restarted, and the bucket has been mounted, it'll fail when mounting again.
I don't find a parameter to mount a mounted path, it'll return 255. Need to find a way to understand the mounted point.

Here we can still use command line, another option may use Alluxio java class but it'll depend on Alluxio jar.
I'll first try command line since it's simple.

GaryShen2008 · 2022-07-04T08:35:04Z

build

GaryShen2008 · 2022-07-05T01:48:37Z

Forgot to consider the case when driver context restarted, and the bucket has been mounted, it'll fail when mounting again. I don't find a parameter to mount a mounted path, it'll return 255. Need to find a way to understand the mounted point.

Here we can still use command line, another option may use Alluxio java class but it'll depend on Alluxio jar. I'll first try command line since it's simple.

Get mounted point by parsing the output of "alluxio fs mount"

GaryShen2008 · 2022-07-05T06:28:49Z

docs/configs.md

-<a name="alluxio.pathsToReplace"></a>spark.rapids.alluxio.pathsToReplace|List of paths to be replaced with corresponding alluxio scheme. Eg, when configureis set to "s3:/foo->alluxio://0.1.2.3:19998/foo,gcs:/bar->alluxio://0.1.2.3:19998/bar", which means:       s3:/foo/a.csv will be replaced to alluxio://0.1.2.3:19998/foo/a.csv and      gcs:/bar/b.csv will be replaced to alluxio://0.1.2.3:19998/bar/b.csv|None
+<a name="alluxio.automount.enabled"></a>spark.rapids.alluxio.automount.enabled|Enable the feature of auto mounting the cloud storage to Alluxio. It requires the Alluxio master is the same node of Spark driver node. When it's true, it requires an environment variable ALLUXIO_HOME be set properly. The Alluxio master's host and port will be read from alluxio.master.hostname and alluxio.master.rpc.port(default: 19998) from ALLUXIO_HOME/conf/alluxio-site.properties, then replace a cloud path which matches spark.rapids.alluxio.bucket.regex like "s3://bar/b.csv" to "alluxio://0.1.2.3:19998/bar/b.csv", and the bucket "s3://bar" will be mounted to "/bar" in Alluxio automatically.|false
+<a name="alluxio.bucket.regex"></a>spark.rapids.alluxio.bucket.regex|A regex to decide which bucket should be auto-mounted to Alluxio. E.g. when setting as "^s3://bucket.*", the bucket which starts with "s3://bucket" will be mounted to Alluxio and the path "s3://bucket-foo/a.csv" will be replaced to "alluxio://0.1.2.3:19998/bucket-foo/a.csv". It's only valid when setting spark.rapids.alluxio.automount.enabled=true. The default value matches all the buckets in "s3://" or "s3a://" scheme.|^s3a{0,1}://.*
+<a name="alluxio.cmd"></a>spark.rapids.alluxio.cmd|Provide the Alluxio command, which is used to mount or get information. E.g. "su,ubuntu,-c,/opt/alluxio-2.8.0/bin/alluxio", it means: run Process(Seq("su", "ubuntu", "-c", "/opt/alluxio-2.8.0/bin/alluxio fs mount --readonly /bucket-foo s3://bucket-foo")), to mount s3://bucket-foo to /bucket-foo. the delimiter "," is used to convert to Seq[String] when you need to use a special user to run the mount command.|None


It allows to define a customized command. We may remove it for a security concern.

wbo4958 · 2022-07-06T05:52:26Z

We'll find the access key and secret key from spark config

Will these configurations display on the Spark UI? and if yes, is it ok for that?

docs/configs.md

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

wbo4958 · 2022-07-06T06:10:27Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+        var alluxio_master: String = null
+        var buffered_source: Source = null
+        try {
+          buffered_source = Source.fromFile(alluxio_home + "/conf/alluxio-site.properties")


here, maybe we can use the withResources in Arm to wrap a closable object

Considered once, but seems not worth to create an AutoCloseable class just for using here.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

wbo4958 · 2022-07-06T06:16:32Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+  // This function will only read once from ALLUXIO/conf.
+  private def initAlluxioInfo(conf: RapidsConf): Unit = {
+    this.synchronized {
+      alluxio_home = scala.util.Properties.envOrElse("ALLUXIO_HOME", "/opt/alluxio-2.8.0")


is there any reason why not to put the alluxio_home/alluxio_Cmd into the if (!isInit) block?

Just consider for the case that user changed the spark.rapids.alluxio.cmd at runtime.
So, it can be read in and use the new command. Mainly easier for debug case.

wbo4958 · 2022-07-06T06:22:03Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+        alluxioMasterHost = Some(alluxio_master + ":" + alluxio_port)
+        // load mounted point by call Alluxio mount command.
+        // We also can get from REST API http://alluxio_master:alluxio_web_port/api/v1/master/info.
+        val (ret, output) = runAlluxioCmd(" fs mount")


I saw some exceptions will be thrown when failed to detect alluxio configuration or something else. But here if the "fs mount" failed to run, do we need to throw the exception also?

Here I want to get the mounted path from alluxio command, but if failed to get, I regard it as no mounted path, and the exception will be threw when mounting a new path later.

its weird to have space in front of "fs"... have the runAlluxioCmd do it if needed

The alluxio command seq is like ("su", "ubuntu", "-c", "/opt/alluxio-2.8.0/bin/alluxio"),
The string of " fs mount" is supposed to append to the last item in the seq to generate like ("su", "ubuntu", "-c", "/opt/alluxio-2.8.0/bin/alluxio fs mount").
The original command should be su ubuntu -c "/opt/alluxio-2.8.0/bin/alluxio fs mount".

Couple of tests done in local workstation,here are my thoughts:

The error is like java.lang.RuntimeException: Mount bucket s3a://mybucket/ to /mybucket failed 1. Here the 1 is the stdout which does not give much debug info. I hope we can print stderr and stdout to show the reason why mount failed.

In my env, there is no need to "su" because the Alluxio cluster and Spark users are the same.
So if I tried to set:

spark.conf.set("spark.rapids.alluxio.cmd", "/home/xxx/alluxio-2.8.0/bin/alluxio")

Then it will fail with :

java.io.IOException: Cannot run program "/home/xxx/alluxio-2.8.0/bin/alluxio fs mount --readonly --option s3a.accessKeyId=xxx --option s3a.secretKey=yyy /mybucket s3a://mybucket/": error=2, No such file or directory

I think the latest code has fixed above issue.

I still think runAlluxioCmd should deal with putting a space in between it rather then having caller do it. we can do it later as followup though too

Changed to add the space in runAlluxioCmd.

@GaryShen2008 I just confirmed that now the issue regarding spark.rapids.alluxio.cmd got fixed based on my test on my local workstation. Now it does not need to su

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

docs/configs.md

docs/get-started/getting-started-alluxio.md

tgravescs · 2022-07-06T13:23:06Z

docs/get-started/getting-started-alluxio.md

+``` shell
+--conf spark.rapids.alluxio.automount.enabled=true
+```
+If Alluxio is not installed in /opt/alluxio-2.8.0, you should set the environment variable `ALLUXIO_HOME`.


It hink we should rephrase this just to state alluxio must be installed and ALLUXIO_HOME must be set to the installation location.

Just hope the user doesn't need to set it explicitly when alluxio is installed in /opt/alluxio-2.8.0.
Otherwise, the user must remember to set it when creating the DB cluster.
More user-friendly, I think.
@viadea What do you think?

I am fine with the current setting.

In the future when the latest alluxio is 2.9 for example, we can update the default value+docs as well.

I removed this line, since now we only read the alluxio command path from the alluxio.cmd config.
The ALLUXIO_HOME is only used for reading Alluxio configurations.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

wbo4958 · 2022-07-08T09:36:12Z

one question,

val df = spark.read.parquet("s3:/bucket1/xxx")
df.show()


val df2 = spark.read.parquet("s3:/bucket2/xxx")
df2.show()

will this PR auto-mount s3:/bucket1 and s3:/bucket2 ?

tgravescs · 2022-07-19T14:25:29Z

is this all working and ready for another review?

GaryShen2008 · 2022-07-22T15:51:44Z

is this all working and ready for another review?

Yes, I think so.

docs/configs.md

docs/get-started/getting-started-alluxio.md

docs/configs.md

tgravescs · 2022-07-22T20:03:21Z

docs/configs.md

@@ -29,7 +29,10 @@ scala> spark.conf.set("spark.rapids.sql.incompatibleOps.enabled", true)

 Name | Description | Default Value
 -----|-------------|--------------
-<a name="alluxio.pathsToReplace"></a>spark.rapids.alluxio.pathsToReplace|List of paths to be replaced with corresponding alluxio scheme. Eg, when configureis set to "s3:/foo->alluxio://0.1.2.3:19998/foo,gcs:/bar->alluxio://0.1.2.3:19998/bar", which means:       s3:/foo/a.csv will be replaced to alluxio://0.1.2.3:19998/foo/a.csv and      gcs:/bar/b.csv will be replaced to alluxio://0.1.2.3:19998/bar/b.csv|None
+<a name="alluxio.automount.enabled"></a>spark.rapids.alluxio.automount.enabled|Enable the feature of auto mounting the cloud storage to Alluxio. It requires the Alluxio master is the same node of Spark driver node. When it's true, it requires an environment variable ALLUXIO_HOME be set properly. The Alluxio master's host and port will be read from alluxio.master.hostname and alluxio.master.rpc.port(default: 19998) from ALLUXIO_HOME/conf/alluxio-site.properties, then replace a cloud path which matches spark.rapids.alluxio.bucket.regex like "s3://bar/b.csv" to "alluxio://0.1.2.3:19998/bar/b.csv", and the bucket "s3://bar" will be mounted to "/bar" in Alluxio automatically.|false


how should one set ALLUXIO_HOME? for instance does it need to be in the spark-env.sh or can be set on command line when launching spark-submit/spark-shell. On yarn cluster mode might need to use spark.yarn.appMasterEnv..... I'm fine with leaving these details about but woudl be nice to put in docs at some point later

Yes, I think both can work for ALLUXIO_HOME. We read it in driver side. So, ALLUXIO_HOME should be set in driver environment. On Databricks, it's added into the environment variables under spark config. Let me update the doc for the suggested way to set it.

tgravescs · 2022-07-22T20:05:49Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+        alluxioMasterHost = Some(alluxio_master + ":" + alluxio_port)
+        // load mounted point by call Alluxio mount command.
+        // We also can get from REST API http://alluxio_master:alluxio_web_port/api/v1/master/info.
+        val (ret, output) = runAlluxioCmd(" fs mount")


I still think runAlluxioCmd should deal with putting a space in between it rather then having caller do it. we can do it later as followup though too

tgravescs · 2022-07-22T20:09:43Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+  // And we'll append --option to set access_key and secret_key if existing.
+  // Suppose the key doesn't exist when using like Databricks's instance profile
+  private def autoMountBucket(scheme: String, bucket: String,
+                      access_key: Option[String],


indentation should be 4 spaces

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

Signed-off-by: Gary Shen <gashen@nvidia.com>

Check both access key and secret Update document to refer to auto mount section Explain more about limitation Use /bucket in mountedBucket to match fs mount output Use camel case to name variable Use URI to parse the fs mount output Signed-off-by: Gary Shen <gashen@nvidia.com>

Use logDebug Write new functions to return the replaceFunc Use URI to parse the scheme and bucket Signed-off-by: Gary Shen <gashen@nvidia.com>

Support to run the alluxio command without su by Process(String) Signed-off-by: Gary Shen <gashen@nvidia.com>

Signed-off-by: Gary Shen <gashen@nvidia.com>

….scala Remove risk log

Update docs Add a space in runAlluxioCmd Signed-off-by: Gary Shen <gashen@nvidia.com>

Signed-off-by: Gary Shen <gashen@nvidia.com>

Fix a bug in scheme replacement Signed-off-by: Gary Shen <gashen@nvidia.com>

correct indent Signed-off-by: Gary Shen <gashen@nvidia.com>

GaryShen2008 · 2022-07-26T08:40:53Z

build

tgravescs · 2022-07-26T13:16:56Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+  // And we'll append --option to set access_key and secret_key if existing.
+  // Suppose the key doesn't exist when using like Databricks's instance profile
+  private def autoMountBucket(scheme: String, bucket: String,
+                                access_key: Option[String],


indentation still off here, shoudl be 4 spaces from left

tgravescs · 2022-07-26T13:20:56Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+  }
+
+  private def genFuncForPathReplacement(replaceMapOption: Option[Map[String, String]]
+                                       ) : Option[Path => Path] = {


spacing off here as well

tgravescs · 2022-07-26T13:21:05Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala

+  }
+
+  private def genFuncForAutoMountReplacement(conf: RapidsConf, relation: HadoopFsRelation,
+                                             alluxioBucketRegex: String) : Option[Path => Path] = {


s;acing off

GaryShen2008 marked this pull request as draft June 28, 2022 06:41

GaryShen2008 force-pushed the alluxio-auto-mount branch 2 times, most recently from eeebe92 to ba58787 Compare June 30, 2022 08:00

GaryShen2008 added the feature request New feature or request label Jun 30, 2022

GaryShen2008 marked this pull request as ready for review June 30, 2022 11:55

GaryShen2008 marked this pull request as draft July 2, 2022 11:45

GaryShen2008 commented Jul 5, 2022

View reviewed changes

GaryShen2008 marked this pull request as ready for review July 6, 2022 05:43

wbo4958 reviewed Jul 6, 2022

View reviewed changes

docs/configs.md Outdated Show resolved Hide resolved

wbo4958 reviewed Jul 6, 2022

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala Show resolved Hide resolved

wbo4958 reviewed Jul 6, 2022

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala Outdated Show resolved Hide resolved

wbo4958 reviewed Jul 6, 2022

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala Outdated Show resolved Hide resolved

wbo4958 reviewed Jul 6, 2022

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala Outdated Show resolved Hide resolved

tgravescs reviewed Jul 6, 2022

View reviewed changes

GaryShen2008 force-pushed the alluxio-auto-mount branch from 43a06c9 to c32d5e7 Compare July 18, 2022 05:07

viadea mentioned this pull request Jul 20, 2022

[BUG] Failed to cast value false to BooleanType for partition column k1 #6026

Closed

tgravescs assigned GaryShen2008 Jul 22, 2022

tgravescs reviewed Jul 22, 2022

View reviewed changes

GaryShen2008 added 18 commits July 26, 2022 11:43

Fix parameter mistake

5d96828

Signed-off-by: Gary Shen <gashen@nvidia.com>

Add log

a39c163

Signed-off-by: Gary Shen <gashen@nvidia.com>

Fix mount command

8ce386c

Signed-off-by: Gary Shen <gashen@nvidia.com>

Use whitespace to split

c15354f

Signed-off-by: Gary Shen <gashen@nvidia.com>

Update docs

b3332f8

Signed-off-by: Gary Shen <gashen@nvidia.com>

Add synchronized to mount command

aaf6249

Signed-off-by: Gary Shen <gashen@nvidia.com>

Update some logs

05673ad

Signed-off-by: Gary Shen <gashen@nvidia.com>

Update docs

d694b0a

Signed-off-by: Gary Shen <gashen@nvidia.com>

Use Properties to read Alluxio config

81aa819

Signed-off-by: Gary Shen <gashen@nvidia.com>

Fix build error

6ba3800

Signed-off-by: Gary Shen <gashen@nvidia.com>

Add empty line to pass mvn verify

1be3cb8

Signed-off-by: Gary Shen <gashen@nvidia.com>

Address comments

02299aa

Use logDebug Write new functions to return the replaceFunc Use URI to parse the scheme and bucket Signed-off-by: Gary Shen <gashen@nvidia.com>

Fix the command without su user

ca97c6b

Support to run the alluxio command without su by Process(String) Signed-off-by: Gary Shen <gashen@nvidia.com>

Don't use URI since s3 path may include space

8998d69

Signed-off-by: Gary Shen <gashen@nvidia.com>

Update sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils…

4dc25a9

….scala Remove risk log

Fix comments

8c74ed8

Update docs Add a space in runAlluxioCmd Signed-off-by: Gary Shen <gashen@nvidia.com>

Fix the indentation

e0174b1

Signed-off-by: Gary Shen <gashen@nvidia.com>

GaryShen2008 force-pushed the alluxio-auto-mount branch 2 times, most recently from cf06c7a to e0174b1 Compare July 26, 2022 06:36

GaryShen2008 added 2 commits July 26, 2022 15:14

Merge branch 'alluxio-auto-mount' into branch-22.08

a4a1851

Fix a bug in scheme replacement Signed-off-by: Gary Shen <gashen@nvidia.com>

Set default value of alluxio.cmd

ec612ec

correct indent Signed-off-by: Gary Shen <gashen@nvidia.com>

tgravescs reviewed Jul 26, 2022

View reviewed changes

tgravescs approved these changes Jul 26, 2022

View reviewed changes

tgravescs merged commit 98f2571 into NVIDIA:branch-22.08 Jul 26, 2022

Add Alluxio auto mount feature #5925

Add Alluxio auto mount feature #5925

Conversation

GaryShen2008 commented Jun 28, 2022 • edited Loading

GaryShen2008 commented Jun 30, 2022 • edited Loading

GaryShen2008 commented Jul 2, 2022

GaryShen2008 commented Jul 4, 2022

GaryShen2008 commented Jul 5, 2022

Choose a reason for hiding this comment

wbo4958 commented Jul 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wbo4958 commented Jul 8, 2022

tgravescs commented Jul 19, 2022

GaryShen2008 commented Jul 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GaryShen2008 commented Jul 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GaryShen2008 commented Jun 28, 2022 •

edited

Loading

GaryShen2008 commented Jun 30, 2022 •

edited

Loading