Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize mvn dependency download scripts #1416

Merged
merged 2 commits into from
Jan 5, 2021

Conversation

NvTimLiu
Copy link
Collaborator

Move definition of the maven repo into 'version-def.sh', then by overwrite the repo URL
it can download either SNAPSHOT cuDF/spark-rapids jars from the internal repo, for the nightly integration tests;
or it can download the pre-release jars on the Sonatype repo, for the release verify integration tests. The default maven repo is internal one for the nightly SNAPSHOT integration tests.

@NvTimLiu
Copy link
Collaborator Author

build

pxLi
pxLi previously approved these changes Dec 17, 2020
@pxLi
Copy link
Collaborator

pxLi commented Dec 17, 2020

Looks good if both pre-merge test and integration tests can pass w/ the script change

@jlowe jlowe changed the title Optimize mvn denpendency download scripts Optimize mvn dependency download scripts Dec 17, 2020
@jlowe jlowe added the build Related to CI / CD or cleanly building label Dec 17, 2020
jlowe
jlowe previously approved these changes Dec 17, 2020
revans2
revans2 previously approved these changes Dec 17, 2020
@NvTimLiu NvTimLiu dismissed stale reviews from revans2, jlowe, and pxLi via 125d6e8 December 18, 2020 02:06
@NvTimLiu
Copy link
Collaborator Author

fix merge conflicts

@NvTimLiu
Copy link
Collaborator Author

build

@pxLi
Copy link
Collaborator

pxLi commented Dec 18, 2020

build

@@ -29,9 +29,15 @@ IFS=$PRE_IFS
CUDF_VER=${CUDF_VER:-"0.18-SNAPSHOT"}
CUDA_CLASSIFIER=${CUDA_CLASSIFIER:-"cuda10-1"}
PROJECT_VER=${PROJECT_VER:-"0.4.0-SNAPSHOT"}
RROJECT_TEST_VER=${PROJECT_TEST_VER:-"0.4.0-SNAPSHOT"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed this typo:

Suggested change
RROJECT_TEST_VER=${PROJECT_TEST_VER:-"0.4.0-SNAPSHOT"}
PROJECT_TEST_VER=${PROJECT_TEST_VER:-"0.4.0-SNAPSHOT"}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jlowe Thanks a lot!

Move definition of the maven repo into 'version-def.sh', then by overwrite the repo URL
it can download either SNAPSHOT cuDF/spark-rapids jars from the internal repo, for the nightly integration tests;
or it can download the pre-release jars on the Sonatype repo, for the release verify integration tests. The default maven repo is internal one for the nightly SNAPSHOT integration tests.

Signed-off-by: Tim Liu <timl@nvidia.com>
Signed-off-by: Tim Liu <timl@nvidia.com>
@NvTimLiu
Copy link
Collaborator Author

build

Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@NvTimLiu in the future, please avoid force-pushing to PRs that are being reviewed. It makes it difficult to see what has changed since the last time a reviewer looked at the PR and often review comments become disconnected from the code. It's better to merge up to the latest branch, and as a bonus Github won't dismiss prior approvals if the merge was clean after the approval.

@NvTimLiu NvTimLiu merged commit 7963309 into NVIDIA:branch-0.4 Jan 5, 2021
@NvTimLiu
Copy link
Collaborator Author

NvTimLiu commented Jan 5, 2021

Looks good, thanks!

@NvTimLiu in the future, please avoid force-pushing to PRs that are being reviewed. It makes it difficult to see what has changed since the last time a reviewer looked at the PR and often review comments become disconnected from the code. It's better to merge up to the latest branch, and as a bonus Github won't dismiss prior approvals if the merge was clean after the approval.

Got it, thanks!

nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
* Optimize mvn denpendency download scripts

Move definition of the maven repo into 'version-def.sh', then by overwrite the repo URL
it can download either SNAPSHOT cuDF/spark-rapids jars from the internal repo, for the nightly integration tests;
or it can download the pre-release jars on the Sonatype repo, for the release verify integration tests. The default maven repo is internal one for the nightly SNAPSHOT integration tests.

Signed-off-by: Tim Liu <timl@nvidia.com>

* fix typo

Signed-off-by: Tim Liu <timl@nvidia.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
* Optimize mvn denpendency download scripts

Move definition of the maven repo into 'version-def.sh', then by overwrite the repo URL
it can download either SNAPSHOT cuDF/spark-rapids jars from the internal repo, for the nightly integration tests;
or it can download the pre-release jars on the Sonatype repo, for the release verify integration tests. The default maven repo is internal one for the nightly SNAPSHOT integration tests.

Signed-off-by: Tim Liu <timl@nvidia.com>

* fix typo

Signed-off-by: Tim Liu <timl@nvidia.com>
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023
* Add files and empty functions

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Implementing histogram-to-percentile

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Implement `percentile_from_histogram`

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Add Java interface

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Input `percentage` as GPU column

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Wrap in list if needed

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Implementing groupby percentile

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Cleanup and docs

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix null issue

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Implement null check

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix typo

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Cleanup

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Cleanup

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Change back to use the less error-prone segmented sort

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix all nulls cases

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Debugging

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix null handling

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix empty input handling

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix access indices

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Fix accumulated counts computation

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* WIP

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Implement input projection for values with frequencies

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* All tests passed!!!

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Cleanup and docs

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

* Rename files and class

* Change docs

* Rename function, rewrite docs, and cleanup

* Rewrite comments

* Handling zero frequency

* Remove too detail comment

* Update src/main/java/com/nvidia/spark/rapids/jni/Histogram.java

* Update src/main/java/com/nvidia/spark/rapids/jni/Histogram.java

* Change error messages

---------

Signed-off-by: Nghia Truong <nghiat@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants