Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a sql-plugin-api module #6565

Open
5 of 9 tasks
gerashegalov opened this issue Sep 15, 2022 · 1 comment
Open
5 of 9 tasks

Create a sql-plugin-api module #6565

gerashegalov opened this issue Sep 15, 2022 · 1 comment
Assignees
@gerashegalov
Labels
task Work required that improves the product but is not user facing tech debt

Comments

@gerashegalov
Copy link
Collaborator

gerashegalov commented Sep 15, 2022 •
edited
Loading

  • Create a new Maven module that houses our public classes (ShimLoader, SQLPlugin, providers, etc,.).
  • Existing sql-plugin module depends on the new module and no longer supports unshimmed classes (those must be in the new module)

Code in the new module must use ShimLoader to access any sql-plugin classes because they are parallel-world, but sql-plugin code can access the unshimmed classes. This also has the advantage of making it much clearer in the source which classes are publicly advertised and which are hidden in parallel-world.

Originally posted by @jlowe in #6555 (review)

Tasks
Beta Give feedback
  1. POC Create a minimum API module with SparkShimProvider API making sure dist is still correct #9138
    build improve task test
    gerashegalov
  2. Restore aggregator being the single dist dependency #9203
    build
  3. Move ProxyShuffleInternalManagerBase to api [databricks] #9506
    tech debt
    gerashegalov
  4. Remove redundant pass-through shuffle manager classes [databricks] #9444
    tech debt
    gerashegalov
  5. Move simple classes to API #9541
    task
  6. Move simple classes after refactoring to sql-plugin-api #9611
    tech debt
    gerashegalov
  9. [BUG] Detangle static compile-time references from public classes. #6556
    bug reliability
@gerashegalov gerashegalov added task Work required that improves the product but is not user facing ? - Needs Triage Need team to review and classify labels Sep 15, 2022
@gerashegalov gerashegalov mentioned this issue Sep 20, 2022
Closed
@sameerz sameerz added tech debt and removed ? - Needs Triage Need team to review and classify labels Sep 20, 2022
@sameerz sameerz mentioned this issue Sep 20, 2022
Open
@gerashegalov gerashegalov mentioned this issue Feb 2, 2023
Open
@gerashegalov gerashegalov mentioned this issue Apr 19, 2023
Merged
@gerashegalov gerashegalov mentioned this issue Aug 25, 2023
Closed
@abellina
Copy link
Collaborator

As part of this issue, would we do away with unshimmed-common-from-spark311.txt? If so, I am wondering if this thread will be resolved #9113 (comment)

@abellina abellina mentioned this issue Aug 25, 2023
Merged
@gerashegalov gerashegalov self-assigned this Aug 28, 2023
@gerashegalov gerashegalov mentioned this issue Sep 2, 2023
Merged
@gerashegalov gerashegalov mentioned this issue Oct 17, 2023
Merged
gerashegalov added a commit that referenced this issue Oct 24, 2023
@gerashegalov
Remove redundant pass-through shuffle manager classes [databricks] (#…
8bd8a6f 
…9444)

Closes #9456
Contributes to #6565 

- additionally remove unsued pom properties
- remove unnecessary unshimmed*txt entries missed in #9506 
- get shuffle manager definition to the point of an easy template, probably can be a shimplify feature

```scala
/*** spark-rapids-shim-json-lines
{"spark": "${buldver}"}
spark-rapids-shim-json-lines ***/
package com.nvidia.spark.rapids.spark${buldver}

import org.apache.spark.SparkConf
import org.apache.spark.shuffle.rapids.ProxyRapidsShuffleInternalManagerBase

/** A shuffle manager optimized for the RAPIDS Plugin for Apache Spark. */
sealed class RapidsShuffleManager(
    conf: SparkConf,
    isDriver: Boolean
) extends ProxyRapidsShuffleInternalManagerBase(conf, isDriver)
```
 
Signed-off-by: Gera Shegalov <gera@apache.org>
gerashegalov added a commit to gerashegalov/spark-rapids that referenced this issue Oct 25, 2023
@gerashegalov
Move RapidsShuffleHeartbeatHandler and messages to API
65824f3 
contributes to NVIDIA#6565

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this issue Oct 25, 2023
Merged
revans2 pushed a commit that referenced this issue Oct 26, 2023
@gerashegalov
Move simple classes to API (#9541)
01a2d20 
contributes to #6565

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov mentioned this issue Nov 2, 2023
Merged
gerashegalov added a commit that referenced this issue Nov 3, 2023
@gerashegalov
Move simple classes after refactoring to sql-plugin-api (#9611)
614f873 
I noticed a couple more classes that can be easily moved to sql-plugin-api after refactoring

Contributes to #6565 

- RapidsUDF
- ExclusiveModeGpuDiscoveryPlugin
- Retryable
- udf.Plugin
- SQLPlugin

to sql-plugin-api

Signed-off-by: Gera Shegalov <gera@apache.org>
@jlowe jlowe mentioned this issue Jul 23, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gerashegalov gerashegalov

Labels
task Work required that improves the product but is not user facing tech debt
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abellina @gerashegalov @sameerz