Cleans up some of the redundant code in proxy/internal RAPIDS Shuffle Managers [databricks]

[abellina]

Signed-off-by: Alessandro Bellina abellina@nvidia.com

Closes #3624.

I am looking to cleanup some of the proxy/base/internal classes in RapidsShuffleManager as per the issue @gerashegalov raised a while back. As it is now, each shim has 3 things:

• RapidsShuffleManager (what the user sets, which subclasses ProxyRapidsShuffleInternalManager)
• ProxyRapidsShuffleInternalManager (which subclasses from ProxyRapidsShuffleInternalManagerBase)
• RapidsShuffleInternalManager (which subclasses from RapidsShuffleInternalManagerBase)

And essentially the RapidsShuffleManager (aka the proxy) loads the internal manager later in time. We have different versions of this so we get one specific to each shim, for binary compatibility reasons.

This change mainly moves getReader back to the RapidsShuffleInternalManagerBase so we can remove some redundant code (this path was the same everywhere I saw). It also renames VisibleShuffleManager to RapidsShuffleManagerLike, but this is not something I feel too strongly about. @gerashegalov could you take a look to see if you have other recommendations that can be done here?

There's more that can be done, for example ShuffleManager is consistent for all the spark versions we support (well not sure about the vendor specific ones as much), so we could in theory have a single RapidsShuffleManager that really just subclasses from ProxyRapidsShuffleInternalManagerBase directly and also implements ShuffleManager... but that seems like a bigger change.

[gerashegalov]

LGTM

[gerashegalov]

build