Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] kvikio v24.10 #490

Merged
merged 44 commits into from
Oct 9, 2024
Merged

[RELEASE] kvikio v24.10 #490

merged 44 commits into from
Oct 9, 2024

Conversation

raydouglass
Copy link
Member

❄️ Code freeze for branch-24.10 and v24.10 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.10 until release (merging of this PR).

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-24.10 into main for the release

raydouglass and others added 30 commits July 19, 2024 15:02
@raydouglass
DOC v24.10 Updates [skip ci]
8f40e29
@GPUtester
Merge pull request #412 from rapidsai/branch-24.08
44b2707 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #414 from rapidsai/branch-24.08
49a08c1 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #415 from rapidsai/branch-24.08
31762ca 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #417 from rapidsai/branch-24.08
f77323d 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #418 from rapidsai/branch-24.08
2f61bb1 
Forward-merge branch-24.08 into branch-24.10
@GPUtester
Merge pull request #419 from rapidsai/branch-24.08
587b36a 
Forward-merge branch-24.08 into branch-24.10
@madsbk
Merge branch-24.08 into branch-24.10
a1b7eb2
@AyodeAwe
Merge pull request #422 from madsbk/branch-24.10-merge-24.08
06db089 
Manual forward-merge: branch 24.10 merge 24.08
@GPUtester
Merge pull request #425 from rapidsai/branch-24.08
232d69c 
Forward-merge branch-24.08 into branch-24.10
@madsbk
Fix missing 24.08 => 24.10 renames (#423)
eb2fe7e 
Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Jake Awe (https://github.com/AyodeAwe)
  - Mike Sarahan (https://github.com/msarahan)

URL: #423
@GPUtester
Merge pull request #431 from rapidsai/branch-24.08
865c881 
Forward-merge branch-24.08 into branch-24.10
@bdice
Improve update-version.sh (#432)
c6835f1 
This PR removes the `UCX_PY` version HTTP call from `update-version.sh` because it is not used.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #432
@jameslamb
Use tool.scikit-build.cmake.version, set scikit-build-core minimum-ve…
317c56c 
…rsion (#430)

Contributes to rapidsai/build-planning#58.

`scikit-build-core==0.10.0` was released today (https://github.com/scikit-build/scikit-build-core/releases/tag/v0.10.0), and wheel-building configurations across RAPIDS are incompatible with it.

This proposes upgrading to that version and fixing configuration here in a way that:

* is compatible with that new `scikit-build-core` version
* takes advantage of the forward-compatibility mechanism (`minimum-version`) that `scikit-build-core` provides, to reduce the risk of needing to do this again in the future

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham

URL: #430
@madsbk
build.sh: --pydevelop (#429)
260eff7 
Adding the `--pydevelop` argument to `build.sh`

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #429
@KyleFromNVIDIA
Update pre-commit hooks (#433)
ab84fdf 
This PR updates pre-commit hooks to the latest versions that are supported without causing style check errors.

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #433
@madsbk
Cython refactoring (#434)
ef2a704 
- Moving Cython's `extern` to where there are used instead of having a single `.pxd` file.
- Split `libkvikio.pyx` into multiple files/modules

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #434
@KyleFromNVIDIA
Fix several issues with wheel builds (#439)
56d859e 
* Ensure correct directory layout in both wheels
* Add CMake entry point to libkvikio wheel
* Require kvikio wheel to use libkvikio wheel
* Exclude unneeded source files from wheel

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)
  - Mike Sarahan (https://github.com/msarahan)
  - Robert Maynard (https://github.com/robertmaynard)
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #439
@madsbk
config_path(): return a reference (#438)
e458be4 
Avoid a string copy

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #438
@madsbk
Speeding up Python tests (#435)
aede17f 
Use `assert_array_equal` to speedup the test suite significantly. 
Also some clean up including splitting `test_read_write` into `test_write` and `test_read`.

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #435
@madsbk
Stream sync before calling cufile (#437)
aa3fd11 
Contrary to most of the non-async CUDA API, cuFile does not have the semantic of being ordered with respect to other work in the null stream. Since this is surprising to most users, we now synchronize on the null stream before calling cuFIle read and write by default. 

See [cuFile IO API Functional Specification](https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html#cufile-io-api-functional-specification) and [CUDA Concurrent Execution between Host and Device](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#concurrent-execution-between-host-and-device).

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #437
@seberg
Remove NumPy <2 pin (#441)
35fb9cf 
This PR removes the NumPy<2 pin. This is desirable for RAPIDS projects that do not have a strong numpy/cupy dependency.
Not doing any math, they don't run into subtle issues, but the pin was still necessary briefly before NumPy 2 was initially released.

(Other RAPIDS projects rely heavily on CuPy/NumPy and need the soon to be released 13.3.0 to function properly, e.g. due to promotion related changes in NumPy.)

Authors:
  - Sebastian Berg (https://github.com/seberg)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - James Lamb (https://github.com/jameslamb)
  - https://github.com/jakirkham

URL: #441
@jameslamb
Drop Python 3.9 support (#443)
8a1a1cd 
Contributes to rapidsai/build-planning#88

Finishes the work of dropping Python 3.9 support.

This project stopped building / testing against Python 3.9 as of rapidsai/shared-workflows#235.
This PR updates configuration and docs to reflect that.

## Notes for Reviewers

### How I tested this

Checked that there were no remaining uses like this:

```shell
git grep -E '3\.9'
git grep '39'
git grep 'py39'
```

And similar for variations on Python 3.8 (to catch things that were missed the last time this was done).

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - https://github.com/jakirkham

URL: #443
@madsbk
Make the bounce buffer size configurable (#447)
0b8525a 
`KVIKIO_BOUNCE_BUFFER_SIZE`

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #447
@KyleFromNVIDIA
Update rapidsai/pre-commit-hooks (#448)
3ce8bcc 
This PR updates rapidsai/pre-commit-hooks to the version 0.4.0.

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #448
@madsbk
Update build.sh (#445)
94edeee 
Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #445
@madsbk
Non-static bounce buffer option (#454)
8676e8d 
closes #451 
closes #452

Also some refactor/clean up

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #454
@bdice
Use nvcomp defaults for algo options. (#450)
0e9ebcb 
This PR updates the nvcomp bindings to use nvcomp's defaults rather than hardcode the default options. The defaults changed in nvcomp 4.0.1, so this is needed for #449.

Authors:
  - Bradley Dice (https://github.com/bdice)
  - https://github.com/jakirkham
  - Vukasin Milovanovic (https://github.com/vuule)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Alexey Kamenev (https://github.com/Alexey-Kamenev)
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Lawrence Mitchell (https://github.com/wence-)

URL: #450
@jameslamb
Add support for Python 3.12, relax numcodecs pin (#458)
3d48f2a 
Contributes to rapidsai/build-planning#40

This PR adds support for Python 3.12.

## Notes for Reviewers

This is part of ongoing work to add Python 3.12 support across RAPIDS.
It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213.

A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all
RAPIDS repos have added Python 3.12 support.

### This will fail until all dependencies have been updates to Python 3.12

CI here is expected to fail until all of this project's upstream dependencies support Python 3.12.

This can be merged whenever all CI jobs are passing.

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #458
@vuule
Update nvcomp to 4.0.1 (#449)
59edda0 
Update the nvCOMP version used for compression/decompression to 4.0.1.

See also:

rapidsai/rapids-cmake#633
rapidsai/cudf#16076

Depends on #450.

Authors:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Bradley Dice (https://github.com/bdice)

URL: #449
kingcrimsontianyu and others added 13 commits September 12, 2024 06:56
@kingcrimsontianyu
Intentionally leak static CUDA resources to avoid crash (part 2) (#462)
9d352ef 
The NVbench application `PARQUET_READER_NVBENCH` in libcudf currently crashes with the segmentation fault. To reproduce:

```
./PARQUET_READER_NVBENCH -d 0 -b 1 --run-once -a io_type=FILEPATH -a compression_type=SNAPPY -a cardinality=0 -a run_length=1
```
 
The root cause is that some (1) `thread_local`  objects on the main thread in `libcudf` and (2) `static` objects in `kvikio` are destroyed after `cudaDeviceReset()` in NVbench and upon program termination. These objects should simply be leaked, since their destructors making CUDA calls upon program termination constitutes UB in CUDA.

This simple PR is the kvikIO side of the fix. The other part is done here rapidsai/cudf#16787.

Authors:
  - Tianyu Liu (https://github.com/kingcrimsontianyu)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)

URL: #462
@jameslamb
remove pytest dependency in kvikio conda recipe (#461)
04663b4 
`kvikio`'s conda recipe only has an `import:` test (basically, makes `conda-build` run `python -c "import kvikio"` after building).

It doesn't ask `conda-build` to run any other tests.

Therefore, its `test:` dependency on `pytest` is unnecessary. This proposes removing it.

## Notes for Reviewers

conda docs on how the `test:` section of a recipe works: https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#test-section

Authors:
  - James Lamb (https://github.com/jameslamb)

Approvers:
  - Ray Douglass (https://github.com/raydouglass)

URL: #461
@bdice
Update to flake8 7.1.1. (#463)
adc2fc4 
We need to update flake8 to fix a false-positive that appears with older flake8 versions on Python 3.12.

Authors:
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - James Lamb (https://github.com/jameslamb)

URL: #463
@jameslamb
Use CI workflow branch 'branch-24.10' again [skip ci] (#465)
81070aa
@madsbk
Skip cufile and nvtx3 when cuda isn't found (#455)
e926fad 
Closes #444

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

URL: #455
@vyasr
Remove now unnecessary variable (#466)
42e0d72 
As of #439 the `FIND_KVIKIO_CPP` variable is largely superfluous. In principle we could still choose to support users who want to build the Python package without having the C++ library already available somewhere, but that use case is vanishingly small now that we have C++ wheels and it would require the developer to intentionally build without isolation and omit the C++ wheel from the environment. For advanced use cases like that it is reasonable to ask the user to handle building the C++ library separately on their own and making it available for the Python package to find. None of the normal build cases that we support (conda, pip, devcontainers, DLFW) require this any longer.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #466
@AyodeAwe
update update-version.sh to use packaging lib (#469)
3e0509f
@madsbk
Fix release-after-main bug in get_primary_cuda_context (#472)
44b3f97 
Closes #442

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #472
@madsbk
nvtx support macos (#474)
35ec784 
Closes #471

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Lawrence Mitchell (https://github.com/wence-)

URL: #474
@madsbk
C++ Test (#440)
2053638 
Implement very basic C++ tests.

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #440
@raydouglass
Add license to kvikio wheel (#481)
03fbb45 
Just adds the symlink to the license file for the `kvikio` wheel.

Authors:
  - Ray Douglass (https://github.com/raydouglass)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #481
@madsbk
Docs cleanup (#480)
e66b83b 
Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Lawrence Mitchell (https://github.com/wence-)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #480
@madsbk
Fix no-cuda build (#484)
5a733bd 
Closes #482

Authors:
  - Mads R. B. Kristensen (https://github.com/madsbk)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Bradley Dice (https://github.com/bdice)

URL: #484
@raydouglass raydouglass requested review from a team as code owners October 4, 2024 19:46
@raydouglass raydouglass requested a review from jameslamb October 4, 2024 19:46
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 0% with 43 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (9bdd064) to head (5a733bd).
Report is 38 commits behind head on main.

Files with missing lines Patch % Lines
python/kvikio/kvikio/defaults.py 0.00% 20 Missing ⚠️
python/kvikio/kvikio/benchmarks/single_node_io.py 0.00% 9 Missing ⚠️
python/kvikio/kvikio/buffer.py 0.00% 7 Missing ⚠️
python/kvikio/kvikio/__init__.py 0.00% 5 Missing ⚠️
python/kvikio/kvikio/cufile.py 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@          Coverage Diff          @@
##            main    #490   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         12      13    +1     
  Lines        893     909   +16     
=====================================
- Misses       893     909   +16

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@raydouglass raydouglass merged commit dcf34cb into main Oct 9, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jameslamb jameslamb Awaiting requested review from jameslamb jameslamb is a code owner automatically assigned from rapidsai/packaging-codeowners

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.