Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests of UDF examples from spark-rapids-examples to premerge tests #4834

Closed
jlowe opened this issue Feb 22, 2022 · 3 comments
Closed

Add tests of UDF examples from spark-rapids-examples to premerge tests #4834

jlowe opened this issue Feb 22, 2022 · 3 comments
Labels
build Related to CI / CD or cleanly building feature request New feature or request wontfix This will not be worked on

Comments

@jlowe
Copy link
Member

jlowe commented Feb 22, 2022

The UDF examples jars have been moved to the spark-rapids-examples repository, but we should be testing against truly external UDFs in our premerge builds to verify we do not break this public API in a PR. The premerge build can install the plugin snapshot jar into the local m2 repository, pull the spark-rapids-examples repo, and build and run the UDF tests from that repo against that installed plugin snapshot. I don't think we need to build the full native builds, as that can be covered in nightly tests. The build and test of the Java-only UDF examples should be quick and have only a small impact on the premerge runtime.

@jlowe jlowe added feature request New feature or request ? - Needs Triage Need team to review and classify build Related to CI / CD or cleanly building labels Feb 22, 2022
@jlowe
Copy link
Member Author

jlowe commented Feb 22, 2022

I think we can get sufficient, relatively timely coverage from the nightly UDF example tests, assuming they are setup against the latest plugin snapshot jar.

@jlowe jlowe closed this as completed Feb 22, 2022
@jlowe jlowe removed the ? - Needs Triage Need team to review and classify label Feb 22, 2022
@res-life
Copy link
Collaborator

@NvTimLiu is checking the nightly pipeline which testing agaist the Snapshot dist jar. For per-merge pipeline, is it necessary?

@jlowe
Copy link
Member Author

jlowe commented Feb 23, 2022

For per-merge pipeline, is it necessary?

I think we can go without for now, as long as the nightly UDF example tests are testing against the latest snapshot version of the plugin.

@sameerz sameerz added the wontfix This will not be worked on label Apr 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building feature request New feature or request wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

3 participants