Design doc: multiple manifests per array

[dcherian]

Ready for 30% feedback. I haven't thought about implementation yet, will look at that by tomorrow lunch time

[kthyng]

@dcherian Is this change related to subchunking within a reference file? I've been trying this out in kerchunk and it has been useful. Also is there a possibility of having multiple representations of the same model output within the same reference file? That is, be able to access the same reference file but use one part of it if accessing a time series and another if accessing a single time step. Thanks for any info!

[dcherian]

Is this change related to subchunking within a reference file?

No, it's about splitting up very large manifests. This "subchunking" is already supported

That is, be able to access the same reference file but use one part of it if accessing a time series and another if accessing a single time step.

Presumably these are separated as different arrays or arrays with the same name but in different groups. So that's already supported.

[kthyng]

it's about splitting up very large manifests

I guess I mean being both User 1 and User 2 from your text, at different times. So if I want to read in a time series at one depth I would access one reference file (User 1), and another if I want to read all of a few time steps (User 2). Is that the idea?

[dcherian]

I guess I mean being both User 1 and User 2 from your text, at different times. So if I want to read in a time series at one depth I would access one reference file (User 1), and another if I want to read all of a few time steps (User 2). Is that the idea?

Ah, this is the "multiply chunked arrays" idea, no that is not the focus here. It is an idea I like a lot though.