Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] hypot SQL function #33

Closed
revans2 opened this issue May 28, 2020 · 3 comments · Fixed by #4592
Closed

[FEA] hypot SQL function #33

revans2 opened this issue May 28, 2020 · 3 comments · Fixed by #4592
Assignees
Labels
feature request New feature or request good first issue Good for newcomers SQL part of the SQL/Dataframe plugin

Comments

@revans2
Copy link
Collaborator

revans2 commented May 28, 2020

Is your feature request related to a problem? Please describe.
would be really great to support the SQl function hypot. We can probably do it initially with just

sqrt((lhs * lhs) + (rhs * rhs))

This might not be good enough with overflows etc and we may need to look a little deeper in how to support it.

@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify SQL part of the SQL/Dataframe plugin labels May 28, 2020
@sameerz sameerz added good first issue Good for newcomers and removed ? - Needs Triage Need team to review and classify labels Oct 6, 2020
@NVnavkumar NVnavkumar self-assigned this Jan 18, 2022
@NVnavkumar
Copy link
Collaborator

There are some equivalent forms that avoid overflow/underflow (see https://en.wikipedia.org/wiki/Pythagorean_addition). Will explore.

@revans2
Copy link
Collaborator Author

revans2 commented Jan 18, 2022

To be clear we have not tested for overflow etc in this at all to know what we can/should do. The Spark code is using the java built in Math.hypot

https://docs.oracle.com/javase/8/docs/api/java/lang/Math.html#hypot-double-double-

Which indicates that it takes care of avoiding intermediate overflow and underflow. I am not sure how critical it is when Spark only supports Doubles as input, but it would be good to know.

@NVnavkumar
Copy link
Collaborator

It does look like the naive algorithm is much more prone to overflow because of the squares, I think we can use:

x * sqrt(1 + (y/x)^2), where x = max(abs(lhs), abs(rhs)) and y = min(abs(lhs), abs(rhs))

@NVnavkumar NVnavkumar added this to the Jan 10 - Jan 28 milestone Jan 21, 2022
@NVnavkumar NVnavkumar linked a pull request Jan 21, 2022 that will close this issue
@sameerz sameerz closed this as completed Jan 30, 2022
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
wjxiz1992 referenced this issue in nvliyuan/yuali-spark-rapids Mar 15, 2024
Enable retry for parquet write to hive

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
wjxiz1992 referenced this issue in nvliyuan/yuali-spark-rapids Apr 7, 2024
Enable retry for parquet write to hive

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
nvliyuan referenced this issue in nvliyuan/yuali-spark-rapids May 6, 2024
Enable retry for parquet write to hive

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request good first issue Good for newcomers SQL part of the SQL/Dataframe plugin
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants