[FEA] Introduce the pylibcudf API and subpackage

[GregoryKimball]

Background

cuDF-python is the common way for Python users to interact with libcudf, the CUDA/C++ computational core for RAPIDS dataframe and database operations. However, cuDF-python is designed to closely correspond with the pandas API, and in the process incurs some semantic overhead to libcudf algorithms. For python applications looking for accelerated dataframe operations and where API matching is not useful, "pylibcudf" provides a direct way for python ecosystem to use libcudf. Pylibcudf also makes libcudf APIs available to the python ecosystem even if they are not supported in pandas (e.g. TEXT).

In addition, we can improve the performance and design of cuDF-python by building on a "pylibcudf" foundation and refactoring extra complexity in cuDF-python's Cython layer.

Example performance

Here is an example of the API design we have in mind for pylibcudf.

    # cv1, cv2 and cv3 are ColumnView objects
    tv = pylibcudf.TableView([cv1, cv2])
    gb = pylibcudf.GroupBy(tv)

    req = pylibcudf.AggregationRequest(cv3, [pylibcudf.GroupbyAggregation.sum()])
    keys, results = gb.aggregate([req])

Here are some draft performance results from our prototype, showing good throughput and low overhead.

[vyasr]

Before we can close this issue we will also need to add comprehensive testing. For now, every pylibcudf API is being developed as a back-end for a cuDF Python API, so the existing Python test suite gives us sufficient coverage. We will want to come back and remedy this gap before actually extracting pylibcudf as a separate package.

[vyasr]

pylibcudf development is now under way. I have created a project board for tracking as well as a number of issues to discuss more specific topics. I am therefore now closing this issue in favor of tracking using those.

[GregoryKimball]

FYI #15162 is the spiritual successor to this issue