Merge branch 'main' into fs_exceptions

.github/ISSUE_TEMPLATE/config.yml

@@ -3,9 +3,9 @@ contact_links:
   - name: ✨ Propose a new major feature
     url: https://github.com/zarr-developers/zarr-specs
     about: A new major feature should be discussed in the Zarr specifications repository.
-  - name: ❓ Discuss something on gitter
-    url: https://gitter.im/zarr-developers/community
-    about: For questions like "How do I do X with Zarr?", you can move to our Gitter channel.
+  - name: ❓ Discuss something on Zulip
+    url: https://ossci.zulipchat.com/
+    about: For questions like "How do I do X with Zarr?", you can move to our Zulip Chat.
   - name: ❓ Discuss something on GitHub Discussions
     url: https://github.com/zarr-developers/zarr-python/discussions
     about: For questions like "How do I do X with Zarr?", you can move to GitHub Discussions.

.github/workflows/minimal.yml

@@ -15,7 +15,7 @@ jobs:
     steps:
     - uses: actions/checkout@v4
     - name: Setup Miniconda
-      uses: conda-incubator/setup-miniconda@v3.0.1
+      uses: conda-incubator/setup-miniconda@v3.0.2
       with:
           channels: conda-forge
           environment-file: environment.yml

.github/workflows/python-package.yml

@@ -42,7 +42,7 @@ jobs:
       with:
         fetch-depth: 0
     - name: Setup Miniconda
-      uses: conda-incubator/setup-miniconda@v3.0.1
+      uses: conda-incubator/setup-miniconda@v3.0.2
       with:
           channels: conda-forge
           python-version: ${{ matrix.python-version }}

.github/workflows/windows-testing.yml

@@ -21,7 +21,7 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
-      - uses: conda-incubator/setup-miniconda@v3.0.1
+      - uses: conda-incubator/setup-miniconda@v3.0.2
         with:
           auto-update-conda: true
           python-version: ${{ matrix.python-version }}

.pre-commit-config.yaml

@@ -8,7 +8,7 @@ default_language_version:
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
     # Ruff version.
-    rev: 'v0.2.1'
+    rev: 'v0.2.2'
     hooks:
     - id: ruff
   - repo: https://github.com/psf/black

README.md

@@ -70,10 +70,10 @@
   </td>
 </tr>
 <tr>
-	<td>Gitter</td>
+	<td>Zulip</td>
 	<td>
-		<a href="https://gitter.im/zarr-developers/community">
-		<img src="https://badges.gitter.im/zarr-developers/community.svg" />
+		<a href="https://ossci.zulipchat.com/">
+		<img src="https://img.shields.io/badge/zulip-join_chat-brightgreen.svg" />
 		</a>
 	</td>
 </tr>

docs/index.rst

@@ -25,7 +25,7 @@ Zarr-Python
 `Installation <installation.html>`_ |
 `Source Repository <https://github.com/zarr-developers/zarr-python>`_ |
 `Issue Tracker <https://github.com/zarr-developers/zarr-python/issues>`_ |
-`Gitter <https://gitter.im/zarr-developers/community>`_
+`Zulip Chat <https://ossci.zulipchat.com/>`_
 
 Zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.
 

docs/release.rst

@@ -18,6 +18,9 @@ Release notes
 Unreleased
 ----------
 
+* Change occurrences of % and format() to f-strings.
+  By :user:`Dimitri Papadopoulos Orfanos <DimitriPapadopoulos>` :issue:`1423`.
+
 * FSStore now raises rather than return bad data.
   By :user:`Martin Durant <martindurant>` and :user:`Ian Carroll <itcarroll>` :issue:`1604`.
 
@@ -65,8 +68,8 @@ Docs
 * Add Norman Rzepka to core-dev team.
   By :user:`Joe Hamman <jhamman>` :issue:`1630`.
 
-* Added section about accessing zip files that are on s3.
-  By :user:`Jeff Peck <jeffpeck10x>` :issue:`1613`.
+* Added section about accessing ZIP archives on s3.
+  By :user:`Jeff Peck <jeffpeck10x>` :issue:`1613`, :issue:`1615`, and :user:`Davis Bennett <d-v-b>` :issue:`1641`.
 
 * Add V3 roadmap and design document.
   By :user:`Joe Hamman <jhamman>` :issue:`1583`.
@@ -160,10 +163,10 @@ Maintenance
   By :user:`Davis Bennett <d-v-b>` :issue:`1462`.
 
 * Style the codebase with ``ruff`` and ``black``.
-  By :user:`Davis Bennett` <d-v-b> :issue:`1459`
+  By :user:`Davis Bennett <d-v-b>` :issue:`1459`
 
 * Ensure that chunks is tuple of ints upon array creation.
-  By :user:`Philipp Hanslovsky` <hanslovsky> :issue:`1461`
+  By :user:`Philipp Hanslovsky <hanslovsky>` :issue:`1461`
 
 .. _release_2.15.0:
 
@@ -551,7 +554,7 @@ Maintenance
   By :user:`Saransh Chopra <Saransh-cpp>` :issue:`1079`.
 
 * Remove option to return None from _ensure_store.
-  By :user:`Greggory Lee <grlee77>` :issue:`1068`.
+  By :user:`Gregory Lee <grlee77>` :issue:`1068`.
 
 * Fix a typo of "integers".
   By :user:`Richard Scott <RichardScottOZ>` :issue:`1056`.
@@ -569,7 +572,7 @@ Enhancements
   Since the format is not yet finalized, the classes and functions are not
   automatically imported into the regular `zarr` name space. Setting the
   `ZARR_V3_EXPERIMENTAL_API` environment variable will activate them.
-  By :user:`Greggory Lee <grlee77>`; :issue:`898`, :issue:`1006`, and :issue:`1007`
+  By :user:`Gregory Lee <grlee77>`; :issue:`898`, :issue:`1006`, and :issue:`1007`
   as well as by :user:`Josh Moore <joshmoore>` :issue:`1032`.
 
 * **Create FSStore from an existing fsspec filesystem**. If you have created
@@ -691,7 +694,7 @@ Enhancements
   higher-level array creation and convenience functions still accept plain
   Python dicts or other mutable mappings for the ``store`` argument, but will
   internally convert these to a ``KVStore``.
-  By :user:`Greggory Lee <grlee77>`; :issue:`839`, :issue:`789`, and :issue:`950`.
+  By :user:`Gregory Lee <grlee77>`; :issue:`839`, :issue:`789`, and :issue:`950`.
 
 * Allow to assign array ``fill_values`` and update metadata accordingly.
   By :user:`Ryan Abernathey <rabernat>`, :issue:`662`.
@@ -838,7 +841,7 @@ Bug fixes
 ~~~~~~~~~
 
 * Fix FSStore.listdir behavior for nested directories.
-  By :user:`Greggory Lee <grlee77>`; :issue:`802`.
+  By :user:`Gregory Lee <grlee77>`; :issue:`802`.
 
 .. _release_2.9.4:
 
@@ -922,7 +925,7 @@ Bug fixes
   By :user:`Josh Moore <joshmoore>`; :issue:`781`.
 
 * avoid NumPy 1.21.0 due to https://github.com/numpy/numpy/issues/19325
-  By :user:`Greggory Lee <grlee77>`; :issue:`791`.
+  By :user:`Gregory Lee <grlee77>`; :issue:`791`.
 
 Maintenance
 ~~~~~~~~~~~
@@ -934,7 +937,7 @@ Maintenance
   By :user:`Elliott Sales de Andrade <QuLogic>`; :issue:`799`.
 
 * TST: add missing assert in test_hexdigest.
-  By :user:`Greggory Lee <grlee77>`; :issue:`801`.
+  By :user:`Gregory Lee <grlee77>`; :issue:`801`.
 
 .. _release_2.8.3:
 

docs/tutorial.rst

@@ -774,7 +774,7 @@ the following code::
 
 Any other compatible storage class could be used in place of
 :class:`zarr.storage.DirectoryStore` in the code examples above. For example,
-here is an array stored directly into a Zip file, via the
+here is an array stored directly into a ZIP archive, via the
 :class:`zarr.storage.ZipStore` class::
 
     >>> store = zarr.ZipStore('data/example.zip', mode='w')
@@ -798,12 +798,12 @@ Re-open and check that data have been written::
            [42, 42, 42, ..., 42, 42, 42]], dtype=int32)
     >>> store.close()
 
-Note that there are some limitations on how Zip files can be used, because items
-within a Zip file cannot be updated in place. This means that data in the array
+Note that there are some limitations on how ZIP archives can be used, because items
+within a ZIP archive cannot be updated in place. This means that data in the array
 should only be written once and write operations should be aligned with chunk
 boundaries. Note also that the ``close()`` method must be called after writing
 any data to the store, otherwise essential records will not be written to the
-underlying zip file.
+underlying ZIP archive.
 
 Another storage alternative is the :class:`zarr.storage.DBMStore` class, added
 in Zarr version 2.2. This class allows any DBM-style database to be used for
@@ -846,7 +846,7 @@ respectively require the `redis-py <https://redis-py.readthedocs.io>`_ and
 `pymongo <https://api.mongodb.com/python/current/>`_ packages to be installed.
 
 For compatibility with the `N5 <https://github.com/saalfeldlab/n5>`_ data format, Zarr also provides
-an N5 backend (this is currently an experimental feature). Similar to the zip storage class, an
+an N5 backend (this is currently an experimental feature). Similar to the ZIP storage class, an
 :class:`zarr.n5.N5Store` can be instantiated directly::
 
     >>> store = zarr.N5Store('data/example.n5')
@@ -1000,12 +1000,13 @@ separately from Zarr.
 
 .. _tutorial_copy:
 
-Accessing Zip Files on S3
-~~~~~~~~~~~~~~~~~~~~~~~~~
+Accessing ZIP archives on S3
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The built-in `ZipStore` will only work with paths on the local file-system, however
-it is also possible to access ``.zarr.zip`` data on the cloud. Here is an example of
-accessing a zipped Zarr file on s3:
+The built-in :class:`zarr.storage.ZipStore` will only work with paths on the local file-system; however
+it is possible to access ZIP-archived Zarr data on the cloud via the `ZipFileSystem <https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/zip.html>`_
+class from ``fsspec``. The following example demonstrates how to access
+a ZIP-archived Zarr group on s3 using `s3fs <https://s3fs.readthedocs.io/en/latest/>`_ and ``ZipFileSystem``:
 
     >>> s3_path = "s3://path/to/my.zarr.zip"
     >>> 
@@ -1014,15 +1015,15 @@ accessing a zipped Zarr file on s3:
     >>> fs = ZipFileSystem(f, mode="r")
     >>> store = FSMap("", fs, check=False)
     >>> 
-    >>> # cache is optional, but may be a good idea depending on the situation
+    >>> # caching may improve performance when repeatedly reading the same data
     >>> cache = zarr.storage.LRUStoreCache(store, max_size=2**28)
     >>> z = zarr.group(store=cache)
 
 This store can also be generated with ``fsspec``'s handler chaining, like so:
 
     >>> store = zarr.storage.FSStore(url=f"zip::{s3_path}",  mode="r")
 
-This can be especially useful if you have a very large ``.zarr.zip`` file on s3
+This can be especially useful if you have a very large ZIP-archived Zarr array or group on s3
 and only need to access a small portion of it.
 
 Consolidating metadata
@@ -1161,7 +1162,7 @@ re-compression, and so should be faster. E.g.::
      └── spam (100,) int64
     >>> new_root['foo/bar/baz'][:]
     array([ 0,  1,  2,  ..., 97, 98, 99])
-    >>> store2.close()  # zip stores need to be closed
+    >>> store2.close()  # ZIP stores need to be closed
 
 .. _tutorial_strings:
 

requirements_dev_numpy.txt

@@ -1,4 +1,4 @@
 # Break this out into a separate file to allow testing against
 # different versions of numpy. This file should pin to the latest
 # numpy version.
-numpy==1.26.1
+numpy==1.26.4

requirements_dev_optional.txt

@@ -3,15 +3,15 @@
 lmdb==1.4.1; sys_platform != 'win32'
 # optional library requirements for Jupyter
 ipytree==0.2.2
-ipywidgets==8.1.1
+ipywidgets==8.1.2
 # optional library requirements for services
 # don't let pyup change pinning for azure-storage-blob, need to pin to older
 # version to get compatibility with azure storage emulator on appveyor (FIXME)
 azure-storage-blob==12.16.0 # pyup: ignore
-redis==5.0.1
+redis==5.0.2
 types-redis
 types-setuptools
-pymongo==4.6.1
+pymongo==4.6.2
 # optional test requirements
 coverage
 pytest-cov==4.1.0

zarr/_storage/absstore.py

@@ -1,9 +1,12 @@
 """This module contains storage classes related to Azure Blob Storage (ABS)"""
 
+from typing import Optional
 import warnings
+
 from numcodecs.compat import ensure_bytes
 from zarr.util import normalize_storage_path
 from zarr._storage.store import _get_metadata_suffix, data_root, meta_root, Store, StoreV3
+from zarr.types import DIMENSION_SEPARATOR
 
 __doctest_requires__ = {
     ("ABSStore", "ABSStore.*"): ["azure.storage.blob"],
@@ -67,7 +70,7 @@ def __init__(
         account_name=None,
         account_key=None,
         blob_service_kwargs=None,
-        dimension_separator=None,
+        dimension_separator: Optional[DIMENSION_SEPARATOR] = None,
         client=None,
     ):
         self._dimension_separator = dimension_separator
@@ -84,7 +87,7 @@ def __init__(
 
             blob_service_kwargs = blob_service_kwargs or {}
             client = ContainerClient(
-                "https://{}.blob.core.windows.net/".format(account_name),
+                f"https://{account_name}.blob.core.windows.net/",
                 container,
                 credential=account_key,
                 **blob_service_kwargs,
@@ -141,7 +144,7 @@ def __getitem__(self, key):
         try:
             return self.client.download_blob(blob_name).readall()
         except ResourceNotFoundError:
-            raise KeyError("Blob %s not found" % blob_name)
+            raise KeyError(f"Blob {blob_name} not found")
 
     def __setitem__(self, key, value):
         value = ensure_bytes(value)
@@ -154,7 +157,7 @@ def __delitem__(self, key):
         try:
             self.client.delete_blob(self._append_path_to_prefix(key))
         except ResourceNotFoundError:
-            raise KeyError("Blob %s not found" % key)
+            raise KeyError(f"Blob {key} not found")
 
     def __eq__(self, other):
         return (

zarr/_storage/store.py

@@ -227,7 +227,7 @@ def _validate_key(self, key: str):
             # TODO: Possibly allow key == ".zmetadata" too if we write a
             #       consolidated metadata spec corresponding to this?
         ):
-            raise ValueError("keys starts with unexpected value: `{}`".format(key))
+            raise ValueError(f"key starts with unexpected value: `{key}`")
 
         if key.endswith("/"):
             raise ValueError("keys may not end in /")

zarr/_storage/v3.py

@@ -3,13 +3,14 @@
 from collections import OrderedDict
 from collections.abc import MutableMapping
 from threading import Lock
-from typing import Union, Dict, Any
+from typing import Union, Dict, Any, Optional
 
 from zarr.errors import (
     MetadataError,
     ReadOnlyError,
 )
 from zarr.util import buffer_size, json_loads, normalize_storage_path
+from zarr.types import DIMENSION_SEPARATOR
 
 from zarr._storage.absstore import ABSStoreV3  # noqa: F401
 from zarr._storage.store import (  # noqa: F401
@@ -224,7 +225,9 @@ def get_partial_values(self, key_ranges):
 
 
 class MemoryStoreV3(MemoryStore, StoreV3):
-    def __init__(self, root=None, cls=dict, dimension_separator=None):
+    def __init__(
+        self, root=None, cls=dict, dimension_separator: Optional[DIMENSION_SEPARATOR] = None
+    ):
         if root is None:
             self.root = cls()
         else:
@@ -569,7 +572,7 @@ def __init__(self, store: StoreLike, metadata_key=meta_root + "consolidated/.zme
         consolidated_format = meta.get("zarr_consolidated_format", None)
         if consolidated_format != 1:
             raise MetadataError(
-                "unsupported zarr consolidated metadata format: %s" % consolidated_format
+                f"unsupported zarr consolidated metadata format: {consolidated_format}"
             )
 
         # decode metadata

zarr/_storage/v3_storage_transformers.py

@@ -8,6 +8,7 @@
 
 from zarr._storage.store import StorageTransformer, StoreV3, _rmdir_from_keys_v3
 from zarr.util import normalize_storage_path
+from zarr.types import DIMENSION_SEPARATOR
 
 
 MAX_UINT_64 = 2**64 - 1
@@ -118,7 +119,7 @@ def _copy_for_array(self, array, inner_store):
         return transformer_copy
 
     @property
-    def dimension_separator(self) -> str:
+    def dimension_separator(self) -> DIMENSION_SEPARATOR:
         assert (
             self._dimension_separator is not None
         ), "dimension_separator is not initialized, first get a copy via _copy_for_array."