Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap sync fs for xarray.to_zarr #2533

Merged
merged 7 commits into from
Jan 22, 2025

Conversation

moradology
Copy link
Contributor

@moradology moradology commented Dec 4, 2024

This PR automatically wraps synchronous filesystems in the RemoteStore.from_url function. This is necessary unless we provide a force_async argument on fsspecs url_to_fs function.

I'm not sure adding more flags (even if we sugar them with default values) is the best move. Another option might be to implement a url_to_asyncfs method that simply wraps url_to_fs and wraps any synchronous filesystems that come out automatically but that, too, would require changes here.

This PR is in service of downstream kerchunk compatibility: fsspec/kerchunk#516

It depends on: fsspec/filesystem_spec#1755

@moradology moradology force-pushed the fix/wrap_sync_in_remote branch 2 times, most recently from 8cd084d to 38a841c Compare December 18, 2024 18:39
@moradology moradology marked this pull request as ready for review December 18, 2024 18:40
@moradology
Copy link
Contributor Author

moradology commented Dec 18, 2024

This fix is implicated in the issue here: #2554. Opening for review. Merge (and CI) is blocked pending a release of fsspec with async wrapper and it may be sensible to wait on that release for a bit to ensure compatibility with other projects as there are a lot of moving pieces across the various libraries being updated

@jhamman
Copy link
Member

jhamman commented Jan 4, 2025

@moradology - is there interest in finishing this up? The proposed changes look good but we'll want a test to make sure this is working as expected.

@moradology
Copy link
Contributor Author

Yeah, definite interest in getting back to this - just at the tail end of a long period of vacation/travel and will have time in the coming week!

@moradology moradology force-pushed the fix/wrap_sync_in_remote branch 2 times, most recently from 563e99d to c7e0f59 Compare January 8, 2025 20:28
@jhamman jhamman requested a review from martindurant January 8, 2025 21:22
@dstansby dstansby added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 9, 2025
@github-actions github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jan 13, 2025
Copy link

codecov bot commented Jan 13, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.27%. Comparing base (eaf5d7a) to head (4b8338f).
Report is 426 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2533      +/-   ##
==========================================
- Coverage   99.98%   90.27%   -9.72%     
==========================================
  Files          38       59      +21     
  Lines       14718     6468    -8250     
==========================================
- Hits        14716     5839    -8877     
- Misses          2      629     +627     
Files with missing lines Coverage Δ
src/zarr/storage/_fsspec.py 84.48% <ø> (ø)

... and 96 files with indirect coverage changes

@martindurant
Copy link
Member

martindurant commented Jan 22, 2025

The kerchunk test suite makes extensive use of xr.to_zarr("memory://..."), which currently doesn't work without this change.

  • I am happy to move the import as requested, to get that done
  • I can fix the test to skip for mindeps

OK to proceed?

cc @jhamman

@martindurant martindurant added the bug Potential issues with the zarr-python library label Jan 22, 2025
@martindurant
Copy link
Member

OK, I have no idea what labeler is or why it's not happy. I added one label ("bug") to this PR to be certain.

@moradology
Copy link
Contributor Author

@martindurant I think I figured the labeler's strange behavior out. Should be fixed once I get eyes+approval on #2747

@dstansby
Copy link
Contributor

I wouldn't worry too much about the broken labeller - this (and other PRs) can definitely be merged before it's fixed (although we should definitely fix it!)

@martindurant martindurant merged commit 0c154c3 into zarr-developers:main Jan 22, 2025
29 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Potential issues with the zarr-python library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants