Use Spark's HybridRowQueue to avoid MemoryConsumer API shim

[jlowe]

#640 added a copy of Spark's HybridRowQueue and supporting classes, but this breaks on Spark 3.2.0 because the MemoryConsumer API changed. The original reasons for adding a copy was listed as visibility problems, but this can be worked around by placing our code in a package with visibiility, and such a package already exists.

[jlowe]

Full disclosure: I was not able to verify this fixed the failing cudf_udf integration test on Spark 3.2, as I am unable to get those tests running in my development environment. @firestarman can you verify these changes or explain how to setup a proper environment for the cudf_udf tests to run?

[jlowe]

build

[tgravescs]

@firestarman were you able to validate?

[jlowe]

I'm going to merge this. The nightly build is still broken, and this is highly likely to fix it. I wasn't able to get the cudf_udf integration test to run properly, but I can guarantee that if the integration test still fails after this is merged that it will fail differently. I'll kick the snapshot build and integration test build early to see if there are any subsequent failures.

[firestarman]

Sorry guys, I missed this. I will pay more attention to review requests. Thanks all for this.