Port all conda recipes to `rattler-build` #18054

gforsyth · 2025-02-20T16:40:04Z

Description

Port all condabuild recipes over to use rattler-build instead.

Contributes to rapidsai/build-planning#47

To satisfy rattler, this changes all the licenses in the pyproject.toml files to the SPDX-compliant Apache-2.0 instead of Apache 2.0

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

bdice

Initial comments attached. Please also see my review on UCXX and apply any relevant requests from that PR to this one, too.

ci/build_python.sh

bdice · 2025-02-20T16:51:19Z

ci/build_python.sh

+                    --channel-priority disabled \
+                    --output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \
+                    --test skip \
+                    -c "${CPP_CHANNEL}" \


We prefer long flags for verbosity wherever possible. Please apply this throughout.

Suggested change

-c "${CPP_CHANNEL}" \

--channel "${CPP_CHANNEL}" \

ci/build_python.sh

conda/recipes/cudf-polars/recipe.yaml

bdice · 2025-02-21T04:23:03Z

conda/recipes/cudf-polars/recipe.yaml

+  run:
+    - python
+    - pylibcudf =${{ version }}
+    - polars >=1.20,<1.23


Make sure this pinning is up to date before merging. These values change frequently.

This is being updated in #18076. Bump it if that merges before this PR.

conda/recipes/libcudf/recipe.yaml

conda/recipes/cudf-polars/recipe.yaml

gforsyth · 2025-02-21T16:03:05Z

Computers are the absolute worst.

Despite RAPIDS_CUDA_VERSION being a string, apparently in the process of splitting and extracting the major version, we're getting an integer instead:

       "context": {
        "version": "25.04.00a223",
        "minor_version": "25.04",
        "cuda_version": "11.8",
        "cuda_major": 11,
        "date_string": 250221,
        "head_rev": "1708a669"
      },

rattler-build has a filter for converting from string to int, but not from int to string.
I also can't reproduce this locally.

gforsyth · 2025-02-21T16:09:31Z

rattler-build has a filter for converting from string to int, but not from int to string.
I also can't reproduce this locally.

Ok, wrapping in quotation marks works. It's very strange that this happens in only some repos, but I'll add "wrap these in strings all the time" to the rattler playbook

bdice · 2025-02-25T19:27:13Z

.github/workflows/pr.yaml

Reminder to revert this before merging.

bdice · 2025-02-25T19:29:16Z

conda/recipes/cudf/recipe.yaml

+    - numba-cuda >=0.2.0,<0.3.0a0
+    - numba >=0.59.1,<0.61.0a0
+    - numpy >=1.23,<3.0a0


#18078 will bump these. Check that PR's status before merging.

bdice · 2025-02-25T19:29:55Z

conda/recipes/cudf/recipe.yaml

+    - pylibcudf =${{ version }}
+    - ${{ pin_compatible("rmm", upper_bound="x.x") }}
+    - fsspec >=0.6.0
+    - if:  cuda_major == "11"


Suggested change

- if: cuda_major == "11"

- if: cuda_major == "11"

conda/recipes/custreamz/recipe.yaml

bdice · 2025-02-25T19:43:27Z

conda/recipes/dask-cudf/recipe.yaml

+  - python:
+      imports:
+        - dask_cudf
+      pip_check: false


Let's circle back to this later. I want to pass pip_check if possible.

This is rapidsai/build-planning#151 JYFI.

bdice · 2025-02-25T19:50:20Z

conda/recipes/libcudf/recipe.yaml

+        - cudf_ROOT="$(realpath ./cpp/build)"
+        - export cudf_ROOT
+        - >
+          ./build.sh -n -v \
+            libcudf libcudf_kafka benchmarks tests \
+            --build_metrics --incl_cache_stats --allgpuarch \
+            --cmake-args=\"-DCMAKE_INSTALL_LIBDIR=lib -DCUDF_ENABLE_ARROW_S3=ON\"


Maybe use a single content: | block as suggested for ucxx.

bdice · 2025-02-25T19:57:55Z

conda/recipes/libcudf/recipe.yaml

+        - cmake ${{ cmake_version }}
+        - ${{ compiler("c") }}
+        - ${{ compiler("cxx") }}
+        - ${{ compiler("cuda") }} =${{ cuda_version }}


Suggested change

- ${{ compiler("cuda") }} =${{ cuda_version }}

- ${{ compiler("cuda") }}

bdice · 2025-02-25T19:59:41Z

conda/recipes/libcudf/recipe.yaml

+        - if: cuda_major != "11"
+          then:
+            - cuda-nvtx-dev


I can't remember why we have a -dev package in run: here. 🤔 I think there is a reason but it escapes me. If we have extra cycles later, let's come back to this.

I think it's a typo (by me) -- the original conda-build recipe just has conda-nvtx in there.

bdice · 2025-02-25T20:04:37Z

conda/recipes/pylibcudf/recipe.yaml

+    - packaging
+  ignore_run_exports:
+    from_package:
+      - if: not (cuda_major == "11")


Seems like we've had this working elsewhere:

Suggested change

- if: not (cuda_major == "11")

- if: cuda_major != "11"

copy-pr-bot · 2025-02-25T22:23:46Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

gforsyth · 2025-02-25T22:26:23Z

/ok to test

bdice

I have only a few leftover comments. However, I think we should merge this soon and work out the remaining items in a follow-up PR wherever we can. Getting this in will help us see any problems, and it'll help with reducing merge conflicts with other work that is going on.

bdice · 2025-02-26T00:37:07Z

conda/recipes/cudf/recipe.yaml

+    content:
+      - ./build.sh cudf


Do we want to make changes like this across all the recipes?

Suggested change

content:

- ./build.sh cudf

content: ./build.sh cudf

or

Suggested change

content:

- ./build.sh cudf

content: |

./build.sh cudf

bdice · 2025-02-26T00:59:08Z

conda/recipes/libcudf/recipe.yaml

+        ./build.sh -n -v \
+            libcudf libcudf_kafka benchmarks tests \
+            --build_metrics --incl_cache_stats --allgpuarch \
+            --cmake-args=\"-DCMAKE_INSTALL_LIBDIR=lib -DCUDF_ENABLE_ARROW_S3=ON\"


I wonder if -DCMAKE_INSTALL_LIBDIR=lib is still necessary. I think that maybe we needed that because rapids-cmake wasn't always conda-aware, and we were getting lib64 instead of lib. I'd like to check the package contents and see if they change if this flag is removed (and see if tests pass).

removed the libdir arg, we'll see what happens

bdice · 2025-02-26T01:01:34Z

conda/recipes/libcudf/recipe.yaml

+  - package:
+      name: libcudf-example
+      version: ${{ version }}
+    build:


How'd we get away without an env: section here? In rapidsai/rmm#1800 it seemed like I needed one...

blind luck, I wager

gforsyth · 2025-02-26T16:21:50Z

/ok to test

vyasr · 2025-02-26T19:37:41Z

rattler-build has a filter for converting from string to int, but not from int to string.
I also can't reproduce this locally.

Ok, wrapping in quotation marks works. It's very strange that this happens in only some repos, but I'll add "wrap these in strings all the time" to the rattler playbook

Is there a minimal "expected" behavior here that we could capture and write up in a rattler issue?

gforsyth · 2025-02-26T19:39:26Z

Is there a minimal "expected" behavior here that we could capture and write up in a rattler issue?

I think it might be:

Given that shells have horrendously inconsistent behavior around types, make everything a string unless someone explicitly calls | int

vyasr · 2025-02-26T20:43:01Z

Is there a minimal "expected" behavior here that we could capture and write up in a rattler issue?

I think it might be:

Given that shells have horrendously inconsistent behavior around types, make everything a string unless someone explicitly calls | int

That seems sensible. Want to write that up as a rattler-build issue?

gforsyth · 2025-02-26T20:44:40Z

That seems sensible. Want to write that up as a rattler-build issue?

will do

gforsyth · 2025-02-26T20:59:35Z

rattler-build issue for context types: prefix-dev/rattler-build#1451

bdice · 2025-02-28T22:55:38Z

conda/recipes/libcudf/recipe.yaml

@@ -20,6 +20,15 @@ cache:
  build:
    script:
      content: |
+
+        # Remove `-fdebug-prefix-map` line from CFLAGS and CXXFLAGS so the


What adds these flags?

both conda-build and rattler-build add them in this file in the build environment:
/etc/conda/activate.d/activate-gxx_linux-64.sh

https://github.com/search?q=repo%3Aconda-forge%2Fctng-compiler-activation-feedstock%20fdebug&type=code

There's a line in there like -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION}

but conda-build uses the recipe name and leaves $PKG_VERSION unset, so it doesn't bust the cache.

upstream issue opened to fix this: prefix-dev/rattler-build#1458

but it might take a bit of time to get a fix so adding the workaround here to get us unblocked.

wolfv · 2025-02-28T23:52:46Z

conda/recipes/cudf/recipe.yaml

@@ -0,0 +1,127 @@
+# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json


If you want, then this line can be ommitted as recipe.yaml is in the schema store now.

gforsyth added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Feb 20, 2025

gforsyth requested review from a team as code owners February 20, 2025 16:40

gforsyth requested a review from msarahan February 20, 2025 16:40

github-actions bot added Python Affects Python cuDF API. cudf.polars Issues specific to cudf.polars pylibcudf Issues specific to the pylibcudf package labels Feb 20, 2025

gforsyth changed the title ~~Add rattler-build recipe for libcudf~~ Port all conda recipes to rattler-build Feb 20, 2025

github-actions bot assigned gforsyth Feb 20, 2025

gforsyth force-pushed the rattler-build branch from 62b6a58 to 69f7b7e Compare February 20, 2025 16:59

bdice reviewed Feb 21, 2025

View reviewed changes

conda/recipes/cudf-polars/recipe.yaml Outdated Show resolved Hide resolved

gforsyth force-pushed the rattler-build branch from 88d20a4 to 1c05304 Compare February 21, 2025 16:17

bdice reviewed Feb 25, 2025

View reviewed changes

github-actions bot assigned bdice Feb 26, 2025

bdice approved these changes Feb 26, 2025

View reviewed changes

gforsyth force-pushed the rattler-build branch from 3c6573e to ac2429e Compare February 26, 2025 15:49

gforsyth force-pushed the rattler-build branch 2 times, most recently from bec4719 to e27c1f3 Compare February 26, 2025 16:36

gforsyth added 25 commits February 28, 2025 09:45

fix(libcudf): make sure cmake is in host for --install

e450344

fix(libcudf): only use libcufile on linux64

a79e922

fix(libcudf): add make (CMAKE_BUILD_PROGRAM not set)

0420e2a

fix(pypkgs): add pip to host

23a65b7

fix(libcudf): move cmake to build

aafe473

fix(custreamz): update license, remove double $

b1e3c92

fix(libcudf): fix overlinking

16d6728

fix(cudf): don't overconstrain cuda-version

f049b95

fix(custreamz): add py_version

258bbe5

fix(pylibcudf): simplify neq syntax

a13d1f3

fix(custreamz): remove unnecessary ignore_run_exports

bb41ea3

refactor(libcudf): multiline yaml block for build script

a4c06fe

fix(dask-cudf): remove extraneous ignore_run_export

ba1eae2

fix(libcudf): remove CMAKE_INSTALL_LIBDIR arg

eb84ed5

refactor(all): use multiline yaml block for all script contents

f7fb713

chore(cudf): update numpy and numba pins

90f33d4

fix(py): set explicit py_version for python outputs

058c466

fix(libcudf): allowlist libnvidia-ml.so.1

cfcf834

chore: restore additional commented-out workflow jobs

42ece0b

fix(libcudf): resolve overlinking error

2d7b95c

fix(cudf-polars): bump to <1.24

e3972e9

fix(python): prepend $CPP_CHANNEL to $RATTLER_CHANNELS

167b310

chore: remove verbose flags

29bcce7

chore: remove secrets and env from pure python packages

5c0caff

fix(libcudf): remove cudf_ROOT argument

8a6e1c4

gforsyth force-pushed the rattler-build branch from 06bb8b0 to 8a6e1c4 Compare February 28, 2025 14:45

gforsyth added 2 commits February 28, 2025 11:55

fix(build_python): channel flag is a separate array element

8413cbe

fix(libcudf): remove cache-busting debug line from CFLAGS and CXXFLAGS

76ce12a

bdice reviewed Feb 28, 2025

View reviewed changes

wolfv reviewed Feb 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port all conda recipes to `rattler-build` #18054

Port all conda recipes to `rattler-build` #18054

gforsyth commented Feb 20, 2025 •

edited

Loading

bdice left a comment

bdice Feb 20, 2025

bdice Feb 21, 2025

bdice Feb 25, 2025 •

edited

Loading

gforsyth commented Feb 21, 2025

gforsyth commented Feb 21, 2025

bdice Feb 25, 2025

bdice Feb 25, 2025

bdice Feb 25, 2025

bdice Feb 25, 2025

vyasr Feb 28, 2025

bdice Feb 25, 2025

bdice Feb 25, 2025

bdice Feb 25, 2025

gforsyth Feb 25, 2025

bdice Feb 25, 2025

copy-pr-bot bot commented Feb 25, 2025

gforsyth commented Feb 25, 2025

bdice left a comment

bdice Feb 26, 2025 •

edited

Loading

bdice Feb 26, 2025

gforsyth Feb 26, 2025

bdice Feb 26, 2025

gforsyth Feb 26, 2025

gforsyth commented Feb 26, 2025

vyasr commented Feb 26, 2025

gforsyth commented Feb 26, 2025

vyasr commented Feb 26, 2025

gforsyth commented Feb 26, 2025

gforsyth commented Feb 26, 2025

bdice Feb 28, 2025

gforsyth Feb 28, 2025 •

edited

Loading

wolfv Feb 28, 2025

	- ${{ compiler("cuda") }} =${{ cuda_version }}
	- ${{ compiler("cuda") }}

		@@ -0,0 +1,127 @@
		# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

Port all conda recipes to rattler-build #18054

Are you sure you want to change the base?

Port all conda recipes to rattler-build #18054

Conversation

gforsyth commented Feb 20, 2025 • edited Loading

Description

Checklist

bdice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdice Feb 25, 2025 • edited Loading

Choose a reason for hiding this comment

gforsyth commented Feb 21, 2025

gforsyth commented Feb 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copy-pr-bot bot commented Feb 25, 2025

gforsyth commented Feb 25, 2025

bdice left a comment

Choose a reason for hiding this comment

bdice Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gforsyth commented Feb 26, 2025

vyasr commented Feb 26, 2025

gforsyth commented Feb 26, 2025

vyasr commented Feb 26, 2025

gforsyth commented Feb 26, 2025

gforsyth commented Feb 26, 2025

Choose a reason for hiding this comment

gforsyth Feb 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Port all conda recipes to `rattler-build` #18054

Port all conda recipes to `rattler-build` #18054

gforsyth commented Feb 20, 2025 •

edited

Loading

bdice Feb 25, 2025 •

edited

Loading

bdice Feb 26, 2025 •

edited

Loading

gforsyth Feb 28, 2025 •

edited

Loading