Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bloop updates - require JDK11 in buildall + docs, build bloop for all targets. #9615

Merged

Conversation

PhilDakin
Copy link
Contributor

Enforces JDK11+ for Bloop builds, and builds Bloop for all targets per #9547.

… targets.

Signed-off-by: Phil Dakin <phil.dakin.1@gmail.com>
@sameerz sameerz added bug Something isn't working build Related to CI / CD or cleanly building labels Nov 2, 2023
build/buildall Outdated Show resolved Hide resolved
build/buildall Outdated Show resolved Hide resolved
CONTRIBUTING.md Outdated Show resolved Hide resolved
Signed-off-by: Phil Dakin <phil.dakin.1@gmail.com>
* Add bloopInstall profile

Signed-off-by: Gera Shegalov <gera@apache.org>

* Add bloopInstall profile

Fixes NVIDIA#9621

Signed-off-by: Gera Shegalov <gera@apache.org>

* add comment for install in scala2.13

Signed-off-by: Gera Shegalov <gera@apache.org>

* simplify

Signed-off-by: Gera Shegalov <gera@apache.org>

* undo unrelated

Signed-off-by: Gera Shegalov <gera@apache.org>

* Apply suggestions from code review

Co-authored-by: PhilDakin <phil.dakin.1@gmail.com>

* review 1

Signed-off-by: Gera Shegalov <gera@apache.org>

* Configurable bloopInstall phase

Signed-off-by: Gera Shegalov <gera@apache.org>

* move bloop config dir to execution conf

Signed-off-by: Gera Shegalov <gera@apache.org>

---------

Signed-off-by: Gera Shegalov <gera@apache.org>
Co-authored-by: PhilDakin <phil.dakin.1@gmail.com>
…able from command line.

Signed-off-by: Phil Dakin <phil.dakin.1@gmail.com>
Signed-off-by: Phil Dakin <phil.dakin.1@gmail.com>
Copy link
Collaborator

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but we need more reviews

build/buildall Outdated Show resolved Hide resolved
build/buildall Show resolved Hide resolved
@PhilDakin
Copy link
Contributor Author

@gerashegalov how do I get more reviews?

@gerashegalov
Copy link
Collaborator

gerashegalov commented Nov 7, 2023

how do I get more reviews?

@PhilDakin we'll just wait for for code owners like @jlowe @tgravescs et al to respond when they get to it

Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me

Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, @PhilDakin!

@jlowe
Copy link
Member

jlowe commented Nov 7, 2023

build

@gerashegalov gerashegalov merged commit d25f1fa into NVIDIA:branch-23.12 Nov 7, 2023
37 checks passed
gerashegalov added a commit to gerashegalov/spark-rapids that referenced this pull request Nov 14, 2023
Follow up to NVIDIA#9615

Signed-off-by: Gera Shegalov <gera@apache.org>
gerashegalov added a commit that referenced this pull request Nov 17, 2023
… mode [databricks] (#9698)

Fixes #9691 

This PR aims at reducing unnecessary IncrementalCompiler output invalidations due to jar updates 

- Update build info only when the HEAD revision changes, or on clean
- Produce the final aggregator jar only if the shade plugin output changed
- Skip generating META-INF/ShimServiceProvider if already exists
- It was not breaking anything but it does not make sense to have -Xsource:2.13 in the 2.12 build
- Replace `skip` by `maven.scaladoc.skip` in docs and scripts as a follow up to #9615 
- fix binary-dedupe to not look outside the target/parallel-world dir
- optional skip for unpack of nightly dependencies, need a general caching solution

Repeated executions

Wall Clock pre-PR | Wall Clock post-PR  | Maven command
| -- | -- | -- |
| `01:10 min` |  `13.004 s` | `mvn package -pl tests -am -Dbuildver=330 \` <br>  `    -Dmaven.scaladoc.skip -DskipTests` |
| `01:23 min` | `25.801 s`  | `mvn package -pl tests -am -Dbuildver=330 \` <br>  `    -Dmaven.scaladoc.skip -Dsuffixes=rapids.CastOpSuite \`<br> `    -Dtests='Cast from string to timestamp'` |
| `01:30 min` | `30.00 s` |  `mvn package -Dbuildver=330 \`<br>`    -Ddist.jar.compress=false -DskipTests -Dmaven.scaladoc.skip` |
|  `48.769 s` | `19.253 s` |  `mvn -T1C package  -pl dist  -am -Dbuildver=330 \`<br> `    -Ddist.jar.compress=false -DskipTests \`<br> `    -Drapids.jni.unpack.skip=true` |             

---------
Signed-off-by: Gera Shegalov <gera@apache.org>
Co-authored-by: Jason Lowe <jlowe@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants