Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add HTTP store support #61

Merged
merged 48 commits into from
Dec 17, 2024
Merged
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
c2ac57b
chore(deps): bump zarrs to 0.18.0
LDeakin Nov 27, 2024
4074328
feat: add HTTP store support
LDeakin Nov 27, 2024
2e8599c
update stubs
LDeakin Nov 27, 2024
f61183e
fix: disallow storage_options for HTTP store
LDeakin Nov 28, 2024
3f2d077
add another todo for store info on codec init
LDeakin Nov 28, 2024
bf4e365
Merge remote-tracking branch 'origin/main' into ld/http_store
LDeakin Nov 29, 2024
390f23b
Merge branch 'main' into ld/http_store
LDeakin Dec 2, 2024
3244bee
refactor store initialisation
LDeakin Dec 6, 2024
95ba17b
try build with latest ring
LDeakin Dec 6, 2024
fa4fca5
try add nasm
LDeakin Dec 6, 2024
38636fd
Revert "try add nasm"
LDeakin Dec 6, 2024
3bf21da
Revert "try build with latest ring"
LDeakin Dec 6, 2024
1bae158
CD: bump maturin to 1.7.8
LDeakin Dec 7, 2024
11f4c87
set python architecture for windows aarch64
LDeakin Dec 7, 2024
4020a7c
Revert "set python architecture for windows aarch64"
LDeakin Dec 7, 2024
08476b3
Set MATURIN_USE_XWIN=1 on windows aarch64
LDeakin Dec 7, 2024
56d69c6
Exclude windows aarch64
LDeakin Dec 7, 2024
96c1a24
fix: store/str type hints
LDeakin Dec 9, 2024
d57e037
fix: get_store to get_store_from_config
LDeakin Dec 9, 2024
86883c5
fix: use match in StoreConfig::extract_bound
LDeakin Dec 9, 2024
5e80adf
fix: elide 'py lifetimes in HTTPStoreConfig
LDeakin Dec 9, 2024
31e9101
fix: remove unneeded return statement
LDeakin Dec 9, 2024
60e867b
fix: address clippy::upper_case_acronyms
LDeakin Dec 9, 2024
63fa081
fix: address clippy::single_match_else
LDeakin Dec 9, 2024
301f6d5
Revert "CD: bump maturin to 1.7.8"
LDeakin Dec 9, 2024
190becd
fix: match on remote store name in StoreConfig::extract_bound
LDeakin Dec 9, 2024
be244e1
fix: remove MATURIN_USE_XWIN
LDeakin Dec 9, 2024
d1e3405
fix: remove zarrs config setup in http tests
LDeakin Dec 9, 2024
dd3ac5b
Merge branch 'main' into ld/http_store
flying-sheep Dec 10, 2024
71f1698
fix: TryFrom<&StoreConfig> for store
LDeakin Dec 10, 2024
0643735
fix: add StoreConfig super class
LDeakin Dec 10, 2024
94806cf
fix: convert local store root to string
LDeakin Dec 10, 2024
9b70af1
fix: change unsupported store to NotImplementedError
LDeakin Dec 10, 2024
0a3bf0f
fix: add docs for HTTP store
LDeakin Dec 10, 2024
6e42799
refactor: move store logic into store module
LDeakin Dec 10, 2024
db7c622
refactor: remove CodecPipelineStore
LDeakin Dec 10, 2024
c288bda
fix: return store directly from opendal_builder_to_sync_store
LDeakin Dec 10, 2024
428710a
fix: use zarrs::ReadableWritableListableStorage
LDeakin Dec 10, 2024
e0704dc
Merge branch 'main' into ld/http_store
LDeakin Dec 10, 2024
fd0c6b4
fix(docs): clarify NotImplementedError
LDeakin Dec 11, 2024
78d9642
Revert store enum changes
flying-sheep Dec 12, 2024
028826e
fmt
flying-sheep Dec 12, 2024
97f9428
update stubs
flying-sheep Dec 12, 2024
41ee02e
wtf
flying-sheep Dec 12, 2024
5873e2a
fix: http store root to endpoint
LDeakin Dec 12, 2024
223c57d
Instantiate Rust containers from Python (#72)
flying-sheep Dec 14, 2024
014b04f
Abstract away store manager
flying-sheep Dec 14, 2024
2296ca1
fix(deps): constrain zarr<=3.0.0b3
LDeakin Dec 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,14 @@ You can then use your `zarr` as normal (with some caveats)!

We export a `ZarrsCodecPipeline` class so that `zarr-python` can use the class but it is not meant to be instantiated and we do not guarantee the stability of its API beyond what is required so that `zarr-python` can use it. Therefore, it is not documented here. We also export two errors, `DiscontiguousArrayError` and `CollapsedDimensionError` that can be thrown in the process of converting to indexers that `zarrs` can understand (see below for more details).

At the moment, we only support local filesystems but intend to support more in the future: https://github.com/ilan-gold/zarrs-python/issues/44
At the moment, we only support a subset of the `zarr-python` stores:

- [x] [LocalStore](https://zarr.readthedocs.io/en/main/_autoapi/zarr/storage/local/index.html) (FileSystem)
- [RemoteStore](https://zarr.readthedocs.io/en/main/_autoapi/zarr/storage/remote/index.html)
- [x] [HTTPFileSystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.http.HTTPFileSystem)

A `NotImplemented` error will be raised if a store is not supported.
We intend to support more stores in the future: https://github.com/ilan-gold/zarrs-python/issues/44.

### Configuration

Expand Down
Loading