You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This was caused because the spark301 plugin jars were on the classpath yet the Spark version being used was Spark 3.1.2. It loos like the dist jar was pulling in the aggregator jar, and the pom specifies it defaults to spark.version=3.0.1.
The incorrect aggregator dependency on the published pom should have been fixed by #4265, but the pom being published for dist snapshots still has the aggregator jars. From the most recent snapshot pom on 1206:
<dependencies>
<dependency>
<groupId>com.nvidia</groupId>
<artifactId>rapids-4-spark-aggregator_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>${spark.version.classifier}</classifier>
<scope>compile</scope>
</dependency>
<!--
manually promoting provided cudf as a direct dependency
-->
<dependency>
<groupId>ai.rapids</groupId>
<artifactId>cudf</artifactId>
<version>${cudf.version}</version>
<classifier>${cuda.version}</classifier>
<scope>compile</scope>
</dependency>
</dependencies>
I checked the rapids4spark-version-info.properties in the dist snapshot jar:
I checked the official 21.12.0 artifact in the staging directory, and that pom is correct. It appears to be specific to snapshot artifacts only. Moving this to 22.02.
A recent spark-rapids-ml test job failed with the following error:
This was caused because the spark301 plugin jars were on the classpath yet the Spark version being used was Spark 3.1.2. It loos like the dist jar was pulling in the aggregator jar, and the pom specifies it defaults to spark.version=3.0.1.
The incorrect aggregator dependency on the published pom should have been fixed by #4265, but the pom being published for dist snapshots still has the aggregator jars. From the most recent snapshot pom on 1206:
I checked the
rapids4spark-version-info.properties
in the dist snapshot jar:which indicates it should have been built with the fix in #4265 yet the published dist pom contains the aggregator jar dependency.
The text was updated successfully, but these errors were encountered: