[FEA] Set a default Alluxio Master IP

[viadea]

Currently we need to get the Alluxio Master IP and then use below Spark RAPIDS config to convert the path:

spark.rapids.alluxio.pathsToReplace="s3://foo->alluxio://RM_hostname:19998/foo,gs://bar->alluxio://RM_hostname:19998/bar"

I wish we can set a default Alluxio Master IP. (Say Master Node's IP) so that users do not need to manually find the Alluxio Master IP.
Eg, spark.rapids.alluxio.masterip=<Master Node's IP >.

[abellina]

I am not entirely sure what that IP should be. Is there a guarantee that the master process is over by the Spark master (I assume that's what you are looking to do).

The config you mention here is verbose, especially since RM_hostname:19998 needs to be repeated for each mount. Could this be put into a different config so that the pathsToReplace one becomes: s3://foo->/foo,gs://bar->/bar or something like this?

[viadea]

I am thinking if we can read the Alluxio conf dir, then we will know the alluxio master IP.
Then set it as default one spark.rapids.alluxio.masterip=<Master Node's IP >. unless users decide to change to a different master node IP.

Then as @abellina mentioned, the pathsToReplace can be much simpler.

[GaryShen2008]

Since we use this config to support auto mount feature by calling "alluxio fs mount", I don't think we can support the case that Alluxio master is not the same node of spark driver node(mostly the spark master in Databricks case).
With this limitation, we can always get the alluxio master IP from alluxio's config file.
@viadea If you agree, I think it's no need to have this config. Instead, I'll have a new config like "spark.rapids.alluxio.automount.enabled" to enable the auto mount feature and get the master from alluxio's config.

[viadea]

@GaryShen2008 I am thinking of some scenario:
If Alluxio master node is not one of the Spark nodes, then we could not find the alluxio's config. So how can we know where is the master? Do you think we can have a config spark.rapids.alluxio.masterip which can override the value?

[GaryShen2008]

@GaryShen2008 I am thinking of some scenario: If Alluxio master node is not one of the Spark nodes, then we could not find the alluxio's config. So how can we know where is the master? Do you think we can have a config spark.rapids.alluxio.masterip which can override the value?

In that case, even the current auto mount feature won't work since we call the command line to mount the folder.
If we want to support such case, we'll have to investigate if it's possible to use Alluxio JDK to mount remotely.