Document a safe unshimming algorithm [skip ci]

[gerashegalov]

It is a guide for developers and it should be further automated as the dist jar verification for correct placement of public classes

Current validation for bitwise-identity is just an approximation.

Signed-off-by: Gera Shegalov gera@apache.org

[gerashegalov]

build

[jlowe]

Thanks for writing up this complicated procedure. For the long-term, would it make sense to:

• Create a new Maven module that houses our public classes (ShimLoader, SQLPlugin, providers, etc,.).
• Existing sql-plugin module depends on the new module and no longer supports unshimmed classes (those must be in the new module)

Code in the new module must use ShimLoader to access any sql-plugin classes because they are parallel-world, but sql-plugin code can access the unshimmed classes. This also has the advantage of making it much clearer in the source which classes are publicly advertised and which are hidden in parallel-world.

[gerashegalov]

Thanks for writing up this complicated procedure. For the long-term, would it make sense to:
Hopefully some complexity will go away once fully automated.

• Create a new Maven module that houses our public classes (ShimLoader, SQLPlugin, providers, etc,.).
• Existing sql-plugin module depends on the new module and no longer supports unshimmed classes (those must be in the new module)

Code in the new module must use ShimLoader to access any sql-plugin classes because they are parallel-world, but sql-plugin code can access the unshimmed classes. This also has the advantage of making it much clearer in the source which classes are publicly advertised and which are hidden in parallel-world.

Definite +1 on having a separate module without dependencies on the rest of the project for the entry point classes.

This doc will be helpful in creating this API module (#6556) . I would also suggest that we look at the sccmap strongly connected components to create more granular packages, so not everything is in the kitchen sinks like com.nvidia.spark.rapids.

Furthermore, based on the tool derived from this doc, we should create another dynamic list of the unshimmed classes that is pushed into the conventional top location of the jar when the entire connected component is within spark3xx-common. I think the fewer classes we have in the parallel-worlds. the more classes will be defined via the parent classloader, and the fewer are the chances of running into the clasloading issues.

[gerashegalov]

build