slow perf with object_store

[dcherian]

Here are two dask performance reports for writing the "public" ERA5 dataset using Coiled:

1. S3
2. GCS

For identical amount of data, with S3 it took 15 minutes, with Google Cloud it took 2.5 hours.

On S3, individual write tasks took 11-12sec. While it takes 55s on Google Cloud

[mpiannucci]

On S3, individual write tasks took 11-12sec. While it takes 55s on Google Cloud

Is this on a single worker? Or is this across many processes that icechunk has been pickled across? We have a hunch this is about the overhead of recreating the object_store + fetching google cloud bearer token on the fly

[dcherian]

Multiple workers, but in theory they should be getting unpickled once per worker, so later tasks should not see this overhead?