Replies: 8 comments 17 replies
-
|
Hi @mdsumner - good question, but to answer it I first need to clarify a few things.
No they aren't. But they are still compatible with the Spec! (at least Icechunk is) That's because the zarr specification is technically not opinionated as to how data is stored on disk. The specification merely states how some arbitrary key-value store interface in any language on any type of storage should behave. That KV store could be the canonical Of course the "native zarr" has become pretty important, as there are lots of tools in various languages which expect this layout of chunks on disk. I myself did not fully understand this distinction until recently, and am often sloppy about it.
That's not a great way to think about it, because using xarray is just an implementation detail of the VirtualiZarr python package. It would be perfectly possible to use some other package that doesn't use xarray to take archival files, extract byte range references, then write them into icechunk. In fact if someone added icechunk support to the kerchunk package that's exactly what you would get. Also whilst the resulting Icechunk store can be read by xarray (via xarray's zarr backend, which calls zarr-python, which calls icechunk's python
Zarr-python reading from Icechunk is already aligned with the v3 standard - it's just not "native zarr" because you're using an Reading icechunk is a bit harder than reading native zarr because the on-disk representation is more complex. But it could still be done, ideally by binding to the canonical icechunk rust crate. For example I have suggested that someone write javascript bindings ("
With this context, your question reduces to "given that icechunk exists, should the community still be trying to shoehorn a virtualization layer into native zarr, or instead just use icechunk for anything virtualized?" See Joe's answer to me asking that question. If somebody does create a non-icechunk layout for virtual stores on disk, virtualizarr could write to that too. We actually did have that for one older layout proposal, but got rid of it because no readers support it.
No - a better way to describe it is that VirtualiZarr uses xarray and kerchunk to help generate (zarr-compatible) Icechunk Stores from pre-existing file formats. |
Beta Was this translation helpful? Give feedback.
-
|
Oh wait we already have an issue to track that VRT idea! |
Beta Was this translation helpful? Give feedback.
-
|
I understand the pickle problem 😃 I had to leave this to think a bit, you're saying the serialization I want is the store, because the new format is so fantastic there's no middle man form. That I can appreciate, but damn I still want an xarray vrt for 15000 netcdf files some times without creating a nested parq dir that is then not the actual netcdf-xarray I waited minutes to create. Thanks 👏👏 I'm slowly getting across this |
Beta Was this translation helpful? Give feedback.
-
|
Maybe one could say, that using the store this way for zarr-like data is partly in response to there being no such facility in xarray. |
Beta Was this translation helpful? Give feedback.
-
Yes that's one way of putting it 😆
We should keep talking about what's achievable here (over in #166).
I prefer to think of it just as the xarray project having successfully avoided scope creep: it's an in-memory analytics data structure, it's not also a file format. |
Beta Was this translation helpful? Give feedback.
-
I didn't say that xarray should have done it - but I was asking for a long time how we can take advantage of the merge/concat machinery within xarray to logically join datasets, persist the work it does. It turns out you figured it out, even though I didn't understand this was the case when you first started the project. So: all good! |
Beta Was this translation helpful? Give feedback.
-
|
@TomNicholas gave a great summary of where vzarr fits in here. I should add, that fsspec's ReferenceFileSystem doesn't need to be tied to zarr/vzarr/icechunk at all. It really is just an IO indirection layer. There have been discussion around whether creating references to the binary chunks within parquet/feather2 files is useful, or perhaps finding line-endings in massive and possibly gzipped CSV files. Those are plausible useful workflows you could do, but in practice the ideas have not found an audience, and zarr has been the only consumer for kerchunk output even before vzarr and icechunk existed. |
Beta Was this translation helpful? Give feedback.
-
|
thank you, lots for me to digest here. I'm still confused about how we can have the manifests abstraction without that being described by the spec. How is an implementor to know that a .json or directory of parquet is a valid source of zarr metadata? (I'm certainly not an experienced implementor of specifications ... fwiw). fsspec is also nice, but it's a defacto standard for one programming language (with analogs in C++ and elsewhere, no grand core spec afaik), I need a lot more experience with it specifically before I can explore much more I think |
Beta Was this translation helpful? Give feedback.
-
|
I also need to be using icechunk more, I just got stuck in the Rust dep but I've got help to move on from that. Is there a docker I could shortcut with though? |
Beta Was this translation helpful? Give feedback.
-
Because the way the store gets you from a key to giving you some bytes of data is outside of the zarr spec, and it's in that un-specced
It doesn't know without reading it all first (luckily the json/parquet metadata is never very big). It's valid if there is some |
Beta Was this translation helpful? Give feedback.
-
Maybe - the icechunk test suite CI apparently uses one. But you should ask there / on the icechunk slack channel on the Earthmover slack. |
Beta Was this translation helpful? Give feedback.
-
|
As far as kerchunk specs go, there is a spec: https://fsspec.github.io/kerchunk/spec.html ; although that isn't enough to tell you that it's readable by zarr. I'm sure this repo has something similar too. Mostly, datasets defined in virtual references are pointed to from other URLs, and indicate what that URL is, rather than some special name or magic bytes as formats of the past have used. |
Beta Was this translation helpful? Give feedback.
-
VirtualiZarr doesn't, because Icechunk has a spec. The thing that tells you it's readable by zarr-python is the python |
Beta Was this translation helpful? Give feedback.
-
|
awesome, all these answers really appreciated |
Beta Was this translation helpful? Give feedback.
-
|
ok thanks, this has been bouncing around in my head that the spec is about the references to bytes and not how those are stored ... it turns out R's {pizzarr} and GDAL both work with them, so somehow I had a blocker in my head about what could work: this one I get complaints about codec "fixedscaleoffset", and in this one about shuffle and fill value: this gives me a lot more to go on. In R use pizzarr: library(pizzarr) ## https://github.com/keller-mark/pizzarr (also need Bioconductor Rarr for some compress algs)
u <- "https://projects.pawsey.org.au/vzarr/NSIDC_SEAICE_PS_N25km.parquet"
z <- pizzarr::HttpStore$new(u)
z$listdir()
g <- zarr_open_group(z)
g$get_item("ICECON")
Error in get_codec(config) : Unknown codec shuffle
that also gives me the message about shuffle, but metadata is fine (I have a lot to explore now) str(z$get_consolidated_metadata())
List of 2
$ metadata :List of 12
..$ .zattrs :List of 50
.. ..$ Conventions : chr "CF-1.6, ACDD-1.3"
.. ..$ acknowledgment : chr "These data are produced by the NASA Cryospheric Science Program within the Earth Sciences Division under the Sc"| __truncated__
.. ..$ cdm_data_type : chr "grid"
.. ..$ citation : chr "DiGirolamo, N. E., C. L. Parkinson, D. J. Cavalieri, P. Gloersen, and H. J. Zwally. 2022, updated yearly. Sea I"| __truncated__
.. ..$ contributor_name : chr "Nicolo E. DiGirolamo, Claire Parkinson, Per Gloersen, H. J. Zwally, Donald Cavalieri, Walter Meier, J. Scott St"| __truncated__
.. ..$ contributor_role : chr "project_scientist, project_scientist, project_scientist, project_scientist, project_scientist, scientist, scien"| __truncated__
.. ..$ coverage_content_type : chr "image"
.. ..$ date_created : chr "2022-06-28"
.. ..$ date_metadata_modified : chr "2022-06-28"
.. ..$ date_modified : chr "2022-06-28"
.. ..$ geospatial_bounds : chr "POLYGON ((-3850000 5850000, 3750000 5850000, 3750000 -5350000, -3850000 -5350000, -3850000 5850000))"
.. ..$ geospatial_bounds_crs : chr "EPSG:3411"
.. ..$ geospatial_lat_max : chr "90.0"
.. ..$ geospatial_lat_min : chr "30.980564"
.. ..$ geospatial_lat_units : chr "degrees_north"
.. ..$ geospatial_lon_max : chr "180.0"
.. ..$ geospatial_lon_min : chr "-180.0"
.. ..$ geospatial_lon_units : chr "degrees_east"
.. ..$ geospatial_x_resolution : chr "25000.00 meters"
.. ..$ geospatial_x_units : chr "meters"
.. ..$ geospatial_y_resolution : chr "25000.00 meters"
.. ..$ geospatial_y_units : chr "meters"
.. ..$ id : chr "10.5067/MPYG15WAA4WX"
... |
Beta Was this translation helpful? Give feedback.
-
|
I'm not really sure what any of that code means without more context, but we have open issues on this repo tracking similar-sounding codec problems. |
Beta Was this translation helpful? Give feedback.
-
|
numcodecs is the python (and cython) implementation of codecs for zarr v2, and thetre are additional codecs from elsewhete such as kerchunk and imagecodecs. They are keyed by a simple name, so there's no way to find one if it isn't implemented in your environment/language. Here is shuffle: https://github.com/zarr-developers/numcodecs/blob/main/numcodecs/shuffle.py#L8 In v3, I believe there is a specific way to name codecs to make them findable, and this all lives in the main spec. |
Beta Was this translation helpful? Give feedback.
-
|
The short story is: if you want to read data with a codec unknown to your zarr implementation, you will need to provide it. The code I linked shows that the algorithms at least are very simple. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! Mostly I'm just excited to see that I was wrong, so included basics that show it works. I'll explore |
Beta Was this translation helpful? Give feedback.
-
|
GDAL has gained capacity to read parquet stores from zip urls, and the shuffle filter: I'll see what can be done for pizzarr, fixedscaleoffset seems to be something I could handle too 😃 |
Beta Was this translation helpful? Give feedback.
-
|
(I'm re-reading and catching up some more as I learn, with notes as I go) thanks! |
Beta Was this translation helpful? Give feedback.
-
As far as I understand, none of the virtualization schemes are actually in the Zarr spec, so it's starting to seem like this is a way of serializing a lazy xarray to a store that borrows structure from Zarr.
Is there scope or ongoing discussions about where this will go? It seems that only Zarr-python
and zarrssupport these stores, and alignment to the v3 standard won't bring those features to other implementations. Any thoughts? @martindurant interested in your take especially 🙏Beta Was this translation helpful? Give feedback.
All reactions