Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use src layout and use hatch for packaging #1592

Merged
merged 6 commits into from
Feb 6, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .git_archival.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
node: $Format:%H$
node-date: $Format:%cI$
describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
ref-names: $Format:%D$
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain why we need this file? Looking at hatch-vcs, I don't see any requirement of this configuration.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file allows users to install Zarr from Git archives without encountering version issues. This is not a hatch-vcs thing, instead this file should've been introduced when setuptools-scm was introduced in Zarr - https://setuptools-scm.readthedocs.io/en/latest/usage/#builtin-mechanisms-for-obtaining-version-numbers. hatch-vcs uses setuptools_scm underneath.

Scientific Python also recommends adding git archival support -

You should also add these two files:

.git_archival.txt:

node: $Format:%H$
node-date: $Format:%cI$
describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
ref-names: $Format:%D$

And .gitattributes (or add this line if you are already using this file):

.git_archival.txt  export-subst

This will allow git archives (including the ones generated from GitHub) to also support versioning.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Thanks for the explanation.

1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
*.py linguist-language=python
*.ipynb linguist-documentation
.git_archival.txt export-subst
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ target/
# Jupyter
.ipynb_checkpoints/

# setuptools-scm
zarr/version.py
# VCS versioning
src/zarr/_version.py

# emacs
*~
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ repos:
rev: v1.7.1
hooks:
- id: mypy
files: zarr
files: src
args: []
additional_dependencies:
- types-redis
Expand Down
2 changes: 1 addition & 1 deletion docs/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,7 @@ of the storage specification that is currently implemented is stored under the

Note that the Zarr test suite includes a data fixture and tests to try and ensure that
data format compatibility is not accidentally broken. See the
:func:`test_format_compatibility` function in the :mod:`zarr.tests.test_storage` module
:func:`test_format_compatibility` function in the :mod:`.tests.test_storage` module
for details.

When to make a release
Expand Down
19 changes: 7 additions & 12 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[build-system]
requires = ["setuptools>=64.0.0", "setuptools-scm>1.5.4"]
build-backend = "setuptools.build_meta"
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"


[project]
Expand Down Expand Up @@ -35,7 +35,7 @@ classifiers = [
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
]
license = { text = "MIT" }
license = "MIT"

[project.optional-dependencies]
jupyter = [
Expand Down Expand Up @@ -69,18 +69,13 @@ exclude_lines = [

[tool.coverage.run]
omit = [
"zarr/meta_v1.py",
"src/zarr/meta_v1.py",
"bench/compress_normal.py",
]

[tool.setuptools]
packages = ["zarr", "zarr._storage", "zarr.tests"]
license-files = ["LICENSE.txt"]

[tool.setuptools_scm]
version_scheme = "guess-next-dev"
local_scheme = "dirty-tag"
write_to = "zarr/version.py"
[tool.hatch]
version.source = "vcs"
build.hooks.vcs.version-file = "src/zarr/_version.py"

[tool.ruff]
line-length = 100
Expand Down
2 changes: 1 addition & 1 deletion zarr/__init__.py → src/zarr/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
ZipStore,
)
from zarr.sync import ProcessSynchronizer, ThreadSynchronizer
from zarr.version import version as __version__
from zarr._version import version as __version__

# in case setuptools scm screw up and find version to be 0.0.0
assert not __version__.startswith("0.0.0")
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion zarr/_storage/v3.py → src/zarr/_storage/v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ def setitems(self, values):
values = {self._normalize_key(key): val for key, val in values.items()}

# initialize the /data/root/... folder corresponding to the array!
# Note: zarr.tests.test_core_v3.TestArrayWithFSStoreV3PartialRead fails
# Note: .tests.test_core_v3.TestArrayWithFSStoreV3PartialRead fails
# without this explicit creation of directories
subdirectories = set(os.path.dirname(v) for v in values.keys())
for subdirectory in subdirectories:
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion zarr/meta.py → src/zarr/meta.py
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ def _encode_storage_transformer_metadata(

@classmethod
def _decode_storage_transformer_metadata(cls, meta: Mapping) -> "StorageTransformer":
from zarr.tests.test_storage_v3 import DummyStorageTransfomer
from tests.test_storage_v3 import DummyStorageTransfomer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is a class from tests imported in user-facing functionality?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't know! see #1601

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see, haha! No worries, I'll rebase this once your PR is merged!

from zarr._storage.v3_storage_transformers import ShardingStorageTransformer

# This might be changed to a proper registry in the future
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion zarr/tests/test_attrs.py → tests/test_attrs.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from zarr.attrs import Attributes
from zarr.storage import KVStore, DirectoryStore
from zarr._storage.v3 import KVStoreV3
from zarr.tests.util import CountingDict, CountingDictV3
from tests.util import CountingDict, CountingDictV3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a bit of surprising change. Is it required? Is this what hatch suggests packages do? Generally, I prefer a relative imports within the tests directory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, this was just a personal preference. Happy to revert it!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's revert it. Happy to discuss this change elsewhere but let's keep this pr scope as small as possible.

from zarr.hierarchy import group


Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
MemoryStoreV3,
SQLiteStoreV3,
)
from zarr.tests.util import have_fsspec
from tests.util import have_fsspec

_VERSIONS = (2, 3) if v3_api_available else (2,)

Expand Down
6 changes: 3 additions & 3 deletions zarr/tests/test_core.py → tests/test_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
BaseStore,
v3_api_available,
)
from .._storage.v3_storage_transformers import ShardingStorageTransformer, v3_sharding_available
from zarr._storage.v3_storage_transformers import ShardingStorageTransformer, v3_sharding_available
from zarr.core import Array
from zarr.errors import ArrayNotFoundError, ContainsGroupError
from zarr.meta import json_loads
Expand Down Expand Up @@ -70,9 +70,9 @@
SQLiteStoreV3,
StoreV3,
)
from zarr.tests.test_storage_v3 import DummyStorageTransfomer
from tests.test_storage_v3 import DummyStorageTransfomer
from zarr.util import buffer_size
from zarr.tests.util import abs_container, skip_test_env_var, have_fsspec, mktemp
from tests.util import abs_container, skip_test_env_var, have_fsspec, mktemp

# noinspection PyMethodMayBeStatic

Expand Down
4 changes: 2 additions & 2 deletions zarr/tests/test_creation.py → tests/test_creation.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@
from zarr._storage.store import v3_api_available
from zarr._storage.v3 import DirectoryStoreV3, KVStoreV3
from zarr.sync import ThreadSynchronizer
from zarr.tests.test_storage_v3 import DummyStorageTransfomer
from zarr.tests.util import mktemp, have_fsspec
from tests.test_storage_v3 import DummyStorageTransfomer
from tests.util import mktemp, have_fsspec


_VERSIONS = (None, 2, 3) if v3_api_available else (None, 2)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
import zarr
from zarr.core import Array
from zarr.storage import DirectoryStore, NestedDirectoryStore, FSStore
from zarr.tests.util import have_fsspec
from tests.util import have_fsspec


needs_fsspec = pytest.mark.skipif(not have_fsspec, reason="needs fsspec")
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion zarr/tests/test_hierarchy.py → tests/test_hierarchy.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@
LRUStoreCacheV3,
)
from zarr.util import InfoReporter, buffer_size
from zarr.tests.util import skip_test_env_var, have_fsspec, abs_container, mktemp
from tests.util import skip_test_env_var, have_fsspec, abs_container, mktemp


_VERSIONS = (2, 3) if v3_api_available else (2,)
Expand Down
2 changes: 1 addition & 1 deletion zarr/tests/test_indexing.py → tests/test_indexing.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
PartialChunkIterator,
)

from zarr.tests.util import CountingDict
from tests.util import CountingDict


def test_normalize_integer_selection():
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion zarr/tests/test_n5.py → tests/test_n5.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import json
import atexit

from zarr.tests.util import have_fsspec
from tests.util import have_fsspec


def test_make_n5_chunk_wrapper():
Expand Down
2 changes: 1 addition & 1 deletion zarr/tests/test_storage.py → tests/test_storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@
)
from zarr.storage import FSStore, rename, listdir
from zarr._storage.v3 import KVStoreV3
from zarr.tests.util import CountingDict, have_fsspec, skip_test_env_var, abs_container, mktemp
from tests.util import CountingDict, have_fsspec, skip_test_env_var, abs_container, mktemp
from zarr.util import ConstantMap, json_dumps


Expand Down
2 changes: 1 addition & 1 deletion zarr/tests/test_storage_v3.py → tests/test_storage_v3.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
StoreV3,
ZipStoreV3,
)
from zarr.tests.util import CountingDictV3, have_fsspec, skip_test_env_var, mktemp
from tests.util import CountingDictV3, have_fsspec, skip_test_env_var, mktemp

# pytest will fail to run if the following fixtures aren't imported here
from .test_storage import StoreTests as _StoreTests
Expand Down
6 changes: 3 additions & 3 deletions zarr/tests/test_sync.py → tests/test_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@
from zarr.sync import ProcessSynchronizer, ThreadSynchronizer

# zarr_version fixture must be imported although not used directly here
from zarr.tests.test_attrs import TestAttributes, zarr_version # noqa
from zarr.tests.test_core import TestArray
from zarr.tests.test_hierarchy import TestGroup
from tests.test_attrs import TestAttributes, zarr_version # noqa
from tests.test_core import TestArray
from tests.test_hierarchy import TestGroup


class TestAttributesWithThreadSynchronizer(TestAttributes):
Expand Down
File renamed without changes.
File renamed without changes.
Loading