Zarr storage in asynchronous code

[jacopoabramo]

I'm working on a project where a set of data acquisition devices are streaming data with ZMQ using an in-process communication protocol. I haven't used zarr so far but I would be interested in using it as a primary storage option.

I've read the issues a bit and it seems that with version 3.0 asyncio is natively supported; I checked the documentation but I couldn't find a relevant example on how this is implemented/how to implement a custom writer supporting asyncio. Is there any available example of streaming chunked data to file using asyncio?

[rabernat]

Welcome @jacopoabramo! Zarr should work well for your use case.

You're correct that there is no narrative documentation yet for the asynchronous API. However, there is API documenation.

You code might look something like this

import zarr.api.asynchronous as zarr

# note that asyncio will have the most benefit with remote storage like S3
store = "s3://bucket/path/to/store"
group = await zarr.group(store)
array = await group.create_array("foo", shape=1_000_000, chunks=100_000)
await array.setitem(slice(200_000), 2)

Hope that helps!

Feel free to share more about your use case.

[jacopoabramo]

Hi, @rabernat, thanks for the quick response.

Roughly speaking, the diagram below shows what I want to accomplish:

Basically, there's a sender (publisher) ZMQ socket (but there could be more as well) that are streaming data encoded to a shared ZMQ proxy device (think of it as a glorified queue) which relays to one/multiple receiver (subscriber) sockets. The idea is that the user may select a specific file type to streaming data to and the specific file writer intervines based on preliminary messages sent by the publisher.

As I said, I'm not very familiar with Zarr; my assumption is that different files can be created by the same file writer (the "group" I presume?), but I want to take in the account the possibility that a user may want to write to a different group.

Since multiple threads for data writing are generally not a great idea (thanks, GIL), I wanted to rely on a single thread with an asyncio event loop that decodes the incoming data and deploy multiple tasks to possibly write to multiple groups.

In this case, everything happens in-memory, and the storage is on-disk; there's no interaction with any remote storage.

I don't know if this would fit the use case of the asynchronous API.

EDIT:

# note that asyncio will have the most benefit with remote storage like S3

Are there performance pitfall for on-disk storage using asyncio?

[rabernat]

Ok...sounds quite complicated. Why are you doing this?

Zarr is for storing ND-array data. Zarr spreads data over multiple different files in storage. The user does not interact directly with the underlying files.

If you just want to store files, and you have your own file format, you probably don't need Zarr. If your application wants to manage its own file reading and writing very explicitly, Zarr is not a good choice.

[jacopoabramo]

Sorry, I explained myself poorly. I ended up mixing wrong nomenclatures. I'll try to explain a little bit further with another draft:

The final use case is where one or multiple sensors are streaming encoded data into the ZMQ queue, and each sensor may have different characteristics (the shape of the ND-array may vary - although my assumption is that will always be 2D-arrays at best, the dtype may be different between all sensors, the acquisition rate may differ, etc.). What I'm trying to understand is: does using the Zarr async api provide any benefit for when multiple data streams are attempting to write to the same endpoint?

Once again, I don't want to manipulate Zarr files directly; I just want to store ND-arrays with different characteristics and acquisition rates efficiently

I think that using the term FileWriter in the draft is wrong. Given your answer, I will probably end up creating a single Zarr writer which stores the ND-array in a newly created zarr.Group directly on disk. And then the data can be simply read-back via zarr again in post-processing/analysis/whatever. Apologies for the confusion

EDIT: I should note that the idea is to record videos of unspecified length, where each frame is captured at a different time point. I do not expect this technique to be very efficient for high speed acquisitions, but I would like to be able to stream to a Zarr storage with at least a precision of hundreds of milliseconds of exposure time, give or take.