Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: improves delete_dir for s3fs-backed FsspecStore #2661

Merged
merged 20 commits into from
Feb 14, 2025
Merged
Changes from 4 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
11ac3d9
Implement asynchronous directory deletion in FsspecStore
carshadi Jan 6, 2025
32d8bc9
Merge branch 'zarr-developers:main' into feat-fsspecstore-bulk-delete
carshadi Jan 6, 2025
bfb4397
Use async batched _rm() for FsspecStore.delete_dir()
carshadi Jan 8, 2025
9812aec
Merge branch 'feat-fsspecstore-bulk-delete' of https://github.com/car…
carshadi Jan 8, 2025
f21f1c2
Suppress allowed exceptions instead of try-except-pass
carshadi Jan 8, 2025
220c28b
Adds note on possibly redundant condition in FsspecStore.delete_dir()
carshadi Jan 8, 2025
ece0f0e
Fix: unpack allowed arguments list
carshadi Jan 8, 2025
a280190
Merge branch 'main' into feat-fsspecstore-bulk-delete
jhamman Jan 10, 2025
bf78caf
Merge branch 'main' into feat-fsspecstore-bulk-delete
jhamman Jan 24, 2025
08dc4f9
Adds tests for FsspecStore.delete_dir
carshadi Jan 24, 2025
6ad9a0a
Update src/zarr/storage/_fsspec.py
carshadi Jan 29, 2025
89e1b5f
Remove supports_listing condition from FsspecStore.delete_dir
carshadi Jan 29, 2025
5e67566
use f-string for url formatting
carshadi Jan 29, 2025
a0214e4
assert `store.fs.asynchronous` instead of `store.fs.async_impl`
carshadi Jan 29, 2025
75ee3a5
Merge remote-tracking branch 'upstream/main' into feat-fsspecstore-bu…
carshadi Jan 29, 2025
aa4a7ff
updates release notes
carshadi Jan 29, 2025
8d44aed
Merge remote-tracking branch 'upstream/main' into feat-fsspecstore-bu…
carshadi Feb 14, 2025
96fb2f6
remove unused import
carshadi Feb 14, 2025
f1d90cc
Explicitly construct wrapped local filesystem for test
carshadi Feb 14, 2025
5c48759
Merge branch 'main' into feat-fsspecstore-bulk-delete
dcherian Feb 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions src/zarr/storage/_fsspec.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from __future__ import annotations

import warnings
import inspect
from typing import TYPE_CHECKING, Any

from zarr.abc.store import ByteRangeRequest, Store
Expand Down Expand Up @@ -261,6 +262,24 @@ async def delete(self, key: str) -> None:
except self.allowed_exceptions:
pass

async def delete_dir(self, prefix: str) -> None:
# docstring inherited
if not self.supports_deletes:
raise NotImplementedError('This method is only available for stores that support deletes.')
if not self.supports_listing:
raise NotImplementedError('This method is only available for stores that support directory listing.')
self._check_writable()

path_to_delete = _dereference_path(self.path, prefix)

if hasattr(self.fs, "_rm") and inspect.iscoroutinefunction(self.fs._rm):
try:
await self.fs._rm(path_to_delete, recursive=True)
except self.allowed_exceptions:
pass
else:
raise NotImplementedError("The store does not support async deletes")

async def exists(self, key: str) -> bool:
# docstring inherited
path = _dereference_path(self.path, key)
Expand Down
Loading