Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate C++ conda recipes and add libcudf-tests package #10326

Merged
merged 28 commits into from
Mar 29, 2022

Conversation

ajschmidt8
Copy link
Member

@ajschmidt8 ajschmidt8 commented Feb 17, 2022

This PR includes the following changes:

  • Adds a libcudf-tests package to the libcudf recipe
    • This is a prerequisite for removing "Project Flash" from our build/CI scripts
    • The libcudf-tests package was added as an additional output to the existing libcudf recipe (which was renamed to libcudf-split)
  • Consolidates remaining C++ recipes into libcudf-split recipe
    • This gets rid of a lot of duplicate code between the recipes and reduces the number of times we have to call conda build in our CI scripts

The result of consolidating the C++ recipes into a single libcudf-split recipe is that one top-level build occurs, which builds all of the C++ components (in conda/recipes/libcudf/build.sh) but does not install them. The installation for each package occurs in the corresponding conda/recipes/libcudf/install_libcudf.sh, conda/recipes/libcudf/install_libcudf_kafka.sh, etc. scripts.

There are some implications to these changes. Namely that any top-level host requirement which includes a run_exports value (i.e. shared libraries) must now be manually specified in the run dependencies of the corresponding outputs package. To help keep things DRY, dependency version specs can be specified in conda/recipes/librmm/conda_build_config.yaml. The exception here is the version spec used for cudatoolkit since that comes from an environment variable in the CI process.

@ajschmidt8 ajschmidt8 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Feb 17, 2022
@github-actions github-actions bot added conda libcudf Affects libcudf (C++/CUDA) code. labels Feb 17, 2022
@ajschmidt8 ajschmidt8 force-pushed the update-recipes branch 5 times, most recently from 6444f75 to 82d6c59 Compare February 23, 2022 20:49
@ajschmidt8 ajschmidt8 force-pushed the update-recipes branch 4 times, most recently from c657f13 to c7f29f8 Compare February 27, 2022 19:35
@codecov
Copy link

codecov bot commented Feb 28, 2022

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.06@516a510). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-22.06   #10326   +/-   ##
===============================================
  Coverage                ?   86.34%           
===============================================
  Files                   ?      140           
  Lines                   ?    22294           
  Branches                ?        0           
===============================================
  Hits                    ?    19249           
  Misses                  ?     3045           
  Partials                ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 516a510...7deedf6. Read the comment docs.

rapids-bot bot pushed a commit to rapidsai/rmm that referenced this pull request Mar 17, 2022
This PR updates the `librmm` `conda` recipe with some learnings from rapidsai/cudf#10326.

Namely that the top-level `build.sh` script is the only feasible approach for consolidating the recipes.

The implication of these changes is that any shared libraries used in the top-level build must now manually be specified as `run` dependencies of the corresponding `outputs` package. To help reduce the amount of duplication of version specifications for these packages, dependency versions can be specified in `conda/recipes/librmm/conda_build_config.yaml`. The exception here is the version spec used for `cudatoolkit` since that comes from an environment variable in the CI process.

Authors:
  - AJ Schmidt (https://github.com/ajschmidt8)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)
  - Jordan Jacobelli (https://github.com/Ethyling)

URL: #997
Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com>
@ajschmidt8 ajschmidt8 marked this pull request as draft March 18, 2022 13:31
@ajschmidt8 ajschmidt8 removed the 5 - DO NOT MERGE Hold off on merging; see PR for details label Mar 18, 2022
@ajschmidt8 ajschmidt8 added non-breaking Non-breaking change and removed non-breaking Non-breaking change labels Mar 22, 2022
@ajschmidt8 ajschmidt8 changed the base branch from branch-22.04 to branch-22.06 March 25, 2022 14:22
@ajschmidt8 ajschmidt8 marked this pull request as ready for review March 28, 2022 20:52
@ajschmidt8 ajschmidt8 changed the title Consolidate C++ conda recipes and add libcudf_tests package Consolidate C++ conda recipes and add libcudf-tests package Mar 29, 2022
@ajschmidt8
Copy link
Member Author

@gpucibot merge

@rapids-bot rapids-bot bot merged commit b926f51 into rapidsai:branch-22.06 Mar 29, 2022
@ajschmidt8 ajschmidt8 deleted the update-recipes branch March 29, 2022 14:47
rapids-bot bot pushed a commit that referenced this pull request Mar 31, 2022
The Build Metrics Report was accidentally disabled in #10326. This will add the flags back to the `build.sh` that are required to generate the report during CI.

Authors:
  - David Wendt (https://github.com/davidwendt)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)

URL: #10562
@vyasr vyasr mentioned this pull request Jul 12, 2022
15 tasks
@bdice bdice mentioned this pull request Dec 13, 2022
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CMake CMake build issue improvement Improvement / enhancement to an existing function Java Affects Java cuDF API. libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants