Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch from conda-build to rattler-build #374

Open
wants to merge 42 commits into
base: branch-0.43
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
fca047d
Switch from `conda-build` to `rattler-build`
gforsyth Feb 13, 2025
df73928
Make sure cmake is available on output host
gforsyth Feb 13, 2025
58eaa03
Check package contents
gforsyth Feb 13, 2025
2e83626
fix `rmm` pinning, add `make`, standardize quotes
gforsyth Feb 13, 2025
2ef072f
Add build environment for ucxx output
gforsyth Feb 13, 2025
0717789
Explicit cuda-cudart-dev version
gforsyth Feb 13, 2025
25a0e70
pin cuda-version in examples host
gforsyth Feb 14, 2025
73b4826
(HACK): force exact pinning in `env.yaml`
gforsyth Feb 14, 2025
f995245
force prepended channel name
gforsyth Feb 14, 2025
952b2eb
revert to sequential solves
gforsyth Feb 14, 2025
e27f89a
Revert "revert to sequential solves"
gforsyth Feb 18, 2025
cca3956
Remove sed hacking (doesn't work anyway)
gforsyth Feb 18, 2025
d6d413f
don't prepend `cuda` to examples build string
gforsyth Feb 18, 2025
1338b4e
add missing run exports to libucxx
gforsyth Feb 18, 2025
c4365d5
fix distributed-ucxx build string
gforsyth Feb 18, 2025
170be1a
remove cuda-version from examples host
gforsyth Feb 18, 2025
d3e28c1
explicit build environments per output
gforsyth Feb 18, 2025
7753854
ignore strict cuda-version pinning in examples
gforsyth Feb 18, 2025
5744f54
explicit run ignore of `cuda-version`
gforsyth Feb 18, 2025
4d2e89f
remove duplicate build env
gforsyth Feb 18, 2025
d6df972
ignore cuda run exports in distributed-ucxx
gforsyth Feb 18, 2025
fe45008
Restore `cmake` to tests output
gforsyth Feb 19, 2025
9eb378b
explicit build env for ucxx
gforsyth Feb 19, 2025
3f5e45f
standardize on ignoring `cuda-version` instead of `compiler('cuda')`
gforsyth Feb 19, 2025
bfa2734
Remove python dir cmake arg
gforsyth Feb 20, 2025
c59b00b
flip condition for cudart-dev
gforsyth Feb 20, 2025
1037b03
Remove unneeded conda channel configuration
gforsyth Feb 20, 2025
0fe315a
inline build.sh
gforsyth Feb 20, 2025
238fb68
Generate noarch `distributed_ucxx`
gforsyth Feb 20, 2025
1bf13d1
ignore run exports `cuda-cudart-dev`
gforsyth Feb 20, 2025
25ee3d4
Don't specify `ignore_run_exports` for `noarch`
gforsyth Feb 20, 2025
27b6260
ignore cudart export from cache-host
gforsyth Feb 20, 2025
603a907
ignore exports from `ucx`, fix py version string
gforsyth Feb 20, 2025
6a49743
disable non-conda workflows
gforsyth Feb 20, 2025
bba2275
ignore exports from `python` for c++ outputs
gforsyth Feb 20, 2025
a1f7fbf
buildstringify python version
gforsyth Feb 21, 2025
1ed2ed3
remove version pin from cuda-cudart-dev
gforsyth Feb 21, 2025
3a1d78b
sort filenames in test section
gforsyth Feb 21, 2025
29eefd0
reorder build and requirements section
gforsyth Feb 21, 2025
ffb43a1
Revert "disable non-conda workflows"
gforsyth Feb 21, 2025
b0cf6cc
fix: move `cmake` to `build`
gforsyth Feb 24, 2025
27e8509
handle overlinking errors and ensure deps aren't overconstrained
gforsyth Feb 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 69 additions & 69 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ jobs:
- changed-files
- checks
- conda-cpp-build
- devcontainer
- docs-build
# - devcontainer
# - docs-build
- conda-cpp-tests
- conda-python-tests
- conda-python-distributed-tests
- wheel-build-libucxx
- wheel-build-ucxx
- wheel-tests-ucxx
- wheel-build-distributed-ucxx
- wheel-tests-distributed-ucxx
# - wheel-build-libucxx
# - wheel-build-ucxx
# - wheel-tests-ucxx
# - wheel-build-distributed-ucxx
# - wheel-tests-distributed-ucxx
- telemetry-setup
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-25.04
Expand Down Expand Up @@ -88,16 +88,16 @@ jobs:
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-25.04
with:
build_type: pull-request
docs-build:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04
with:
build_type: pull-request
node_type: "gpu-l4-latest-1"
arch: "amd64"
container_image: "rapidsai/ci-conda:latest"
run_script: "ci/build_docs.sh"
# docs-build:
# needs: conda-cpp-build
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-25.04
# with:
# build_type: pull-request
# node_type: "gpu-l4-latest-1"
# arch: "amd64"
# container_image: "rapidsai/ci-conda:latest"
# run_script: "ci/build_docs.sh"
conda-cpp-tests:
needs: [conda-cpp-build, changed-files]
secrets: inherit
Expand All @@ -123,58 +123,58 @@ jobs:
build_type: pull-request
script: "ci/test_python_distributed.sh"
container-options: "--cap-add CAP_SYS_PTRACE --shm-size=8g --ulimit=nofile=1000000:1000000"
wheel-build-libucxx:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04
with:
build_type: pull-request
script: ci/build_wheel_libucxx.sh
# build for every combination of arch and CUDA version, but only for the latest Python
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
wheel-build-ucxx:
needs: wheel-build-libucxx
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04
with:
build_type: pull-request
script: ci/build_wheel_ucxx.sh
wheel-tests-ucxx:
needs: [wheel-build-ucxx, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
build_type: pull-request
container-options: "--cap-add CAP_SYS_PTRACE --shm-size=8g --ulimit=nofile=1000000:1000000"
script: ci/test_wheel_ucxx.sh
wheel-build-distributed-ucxx:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04
with:
build_type: pull-request
script: ci/build_wheel_distributed_ucxx.sh
wheel-tests-distributed-ucxx:
needs: [wheel-build-ucxx, wheel-build-distributed-ucxx, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
build_type: pull-request
container-options: "--cap-add CAP_SYS_PTRACE --shm-size=8g --ulimit=nofile=1000000:1000000"
script: ci/test_wheel_distributed_ucxx.sh
devcontainer:
secrets: inherit
needs: [checks, telemetry-setup]
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-25.04
with:
arch: '["amd64"]'
cuda: '["12.8"]'
build_command: |
sccache -z;
build-all --verbose;
sccache -s;
# wheel-build-libucxx:
# needs: checks
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04
# with:
# build_type: pull-request
# script: ci/build_wheel_libucxx.sh
# # build for every combination of arch and CUDA version, but only for the latest Python
# matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
# wheel-build-ucxx:
# needs: wheel-build-libucxx
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04
# with:
# build_type: pull-request
# script: ci/build_wheel_ucxx.sh
# wheel-tests-ucxx:
# needs: [wheel-build-ucxx, changed-files]
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04
# if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
# with:
# build_type: pull-request
# container-options: "--cap-add CAP_SYS_PTRACE --shm-size=8g --ulimit=nofile=1000000:1000000"
# script: ci/test_wheel_ucxx.sh
# wheel-build-distributed-ucxx:
# needs: checks
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-25.04
# with:
# build_type: pull-request
# script: ci/build_wheel_distributed_ucxx.sh
# wheel-tests-distributed-ucxx:
# needs: [wheel-build-ucxx, wheel-build-distributed-ucxx, changed-files]
# secrets: inherit
# uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04
# if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
# with:
# build_type: pull-request
# container-options: "--cap-add CAP_SYS_PTRACE --shm-size=8g --ulimit=nofile=1000000:1000000"
# script: ci/test_wheel_distributed_ucxx.sh
# devcontainer:
# secrets: inherit
# needs: [checks, telemetry-setup]
# uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-25.04
# with:
# arch: '["amd64"]'
# cuda: '["12.8"]'
# build_command: |
# sccache -z;
# build-all --verbose;
# sccache -s;

telemetry-summarize:
# This job must use a self-hosted runner to record telemetry traces.
Expand Down
23 changes: 18 additions & 5 deletions ci/build_cpp.sh
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
#!/bin/bash

# SPDX-FileCopyrightText: Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES.
# SPDX-FileCopyrightText: Copyright (c) 2022-2025, NVIDIA CORPORATION & AFFILIATES.
# SPDX-License-Identifier: BSD-3-Clause

set -euo pipefail

rapids-configure-conda-channels

source rapids-configure-sccache

source rapids-date-string
Expand All @@ -19,9 +17,24 @@ rapids-logger "Begin C++ and Python builds"

sccache --zero-stats

RAPIDS_PACKAGE_VERSION=$(rapids-generate-version) rapids-conda-retry mambabuild \
conda/recipes/ucxx
RAPIDS_PACKAGE_VERSION=$(rapids-generate-version)
export RAPIDS_PACKAGE_VERSION

source rapids-rattler-channel-string

# --no-build-id allows for caching with `sccache`
# more info is available at
# https://rattler.build/latest/tips_and_tricks/#using-sccache-or-ccache-with-rattler-build
rattler-build build --recipe conda/recipes/ucxx \
--experimental \
--no-build-id \
--channel-priority disabled \
--output-dir "$RAPIDS_CONDA_BLD_OUTPUT_DIR" \
"${RATTLER_CHANNELS[@]}"

sccache --show-adv-stats

# remove build_cache directory
rm -rf "$RAPIDS_CONDA_BLD_OUTPUT_DIR"/build_cache
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like something that should be an ignore pattern in our upload script instead of something that every build script has to include.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. We opened an issue upstream, too, to add a CLI option to clear the build-cache post-build: prefix-dev/rattler-build#1424


rapids-upload-conda-to-s3 cpp
7 changes: 0 additions & 7 deletions conda/recipes/ucxx/build.sh

This file was deleted.

6 changes: 0 additions & 6 deletions conda/recipes/ucxx/build_and_install_distributed_ucxx.sh

This file was deleted.

7 changes: 0 additions & 7 deletions conda/recipes/ucxx/build_and_install_ucxx.sh

This file was deleted.

6 changes: 0 additions & 6 deletions conda/recipes/ucxx/install_libucxx_examples.sh

This file was deleted.

6 changes: 0 additions & 6 deletions conda/recipes/ucxx/install_libucxx_tests.sh

This file was deleted.

Loading
Loading