Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] buildall --profile=321 fails on missing spark301 rapids-4-spark-sql dependency #4736

Closed
gerashegalov opened this issue Feb 9, 2022 · 0 comments · Fixed by #4738
Closed

[BUG] buildall --profile=321 fails on missing spark301 rapids-4-spark-sql dependency #4736

gerashegalov opened this issue Feb 9, 2022 · 0 comments · Fixed by #4738
Assignees
@gerashegalov
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Milestone
Jan 31 - Feb 11

Comments

@gerashegalov
Copy link
Collaborator

gerashegalov commented Feb 9, 2022 •
edited
Loading

Describe the bug
When building on a clean m2 cache buildall invoked with a list of shims excluding 301 still hits unresolved spark301 dependency. It appears to be related to the behavior of invoking mvn -pl dist without requiring the dependencies built in the same invokation via -am preventing from spark.version property initialized properly.

Steps/Code to reproduce bug

  1. rm -rf ~/.m2/repository
  2. ./build/buildall --profile=321

Expected behavior
buildall is designed to be self-contained and should not resolve versions other than specified via --profile=311,321,...

Environment details (please complete the following information)
local dev on macOS and Linux

Additional context
N/A
@gerashegalov gerashegalov added bug Something isn't working build Related to CI / CD or cleanly building labels Feb 9, 2022
@gerashegalov gerashegalov self-assigned this Feb 9, 2022
@gerashegalov gerashegalov added this to the Jan 31 - Feb 11 milestone Feb 9, 2022
gerashegalov added a commit to gerashegalov/spark-rapids that referenced this issue Feb 9, 2022
@gerashegalov
Change resume-from to aggregator in buildall
4485d4c 
Closes NVIDIA#4736

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this issue Feb 9, 2022
Merged
gerashegalov added a commit that referenced this issue Feb 9, 2022
@gerashegalov
Change resume-from to aggregator in buildall [skip ci] (#4738)
783f841 
Ensure proper spark version classifier profile activation by making aggregator the start of resume-from in the second single-threaded part of the build.

Note that this is a quick fix without much investigation if this is "the right" fix but it works, and should be acceptable for a non-production script. The impact on the build time is negligible

Closes #4736

Signed-off-by: Gera Shegalov gera@apache.org
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerashegalov gerashegalov

Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
Milestone
Jan 31 - Feb 11
Development

Successfully merging a pull request may close this issue.

Change resume-from to aggregator in buildall [skip ci] gerashegalov/spark-rapids
1 participant
@gerashegalov