FAQ question: "Can my dataset be virtualized?"

[TomNicholas]

The restrictions of regular-length chunks and consistent codecs are so important that we should add it to the FAQ. Lots of people keep asking me effectively: "can my dataset be Virtualized" and the FAQ should answer that question explicitly.

(Inspired by the conversation with ITS_LIVE cc @betolink @sharkinsspatial @abarciauskas-bgse )

[abarciauskas-bgse]

Great idea!

[maxrjones]

I like the idea of a FAQ on this as well! This question was also an inspiration for developmentseed/ndquirk#1 recognizing that it can be hard to tell if there are consistent chunks (e.g., MUR SST) so it'd be awesome if people could drop a list of files into an API to enumerate any barriers to virtualization.

[TomNicholas]

drop a list of files into an API to enumerate any barriers to virtualization.

Isn't that kind of what VirtualiZarr is though? Especially once open_virtual_mfdataset is merged. To check that files in object storage can be virtualized you need to do basically the same set of steps that VirtualiZarr does...

[maxrjones]

drop a list of files into an API to enumerate any barriers to virtualization.

Isn't that kind of what VirtualiZarr is though? Especially once open_virtual_mfdataset is merged. To check that files in object storage can be virtualized you need to do basically the same set of steps that VirtualiZarr does...

Yeah there's overlap but I imagine virtualizarr will mostly be used for L3/4 data whereas I'd like for ndquirk to also identify quirkiness in L1/2 data. I haven't started on a design doc yet but an explicit sub-goal is to leverage existing libraries which would ideally include open_virtual_mfdataset. Though off-the-cuff it might be slower to actually construct virtual datasets vs. leaving off that last step due to loadable_variables.