failures writing to tigris

[dcherian]

I've been struggling with writing to Tigris from coiled.

Now with v0.1.2 we have an error message!

File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/array.py", line 2523, in __setitem__
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/_compat.py", line 43, in inner_f
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/array.py", line 2979, in set_orthogonal_selection
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/sync.py", line 142, in sync
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/sync.py", line 98, in _runner
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/array.py", line 1413, in _set_selection
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/codec_pipeline.py", line 468, in write
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/common.py", line 68, in concurrent_map
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/common.py", line 66, in run
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/codec_pipeline.py", line 418, in write_batch
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/common.py", line 68, in concurrent_map
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/common.py", line 66, in run
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/core/codec_pipeline.py", line 416, in _write_key
  File "/opt/coiled/env/lib/python3.12/site-packages/zarr/storage/_common.py", line 144, in set
  File "/opt/coiled/env/lib/python3.12/site-packages/icechunk/store.py", line 206, in set
icechunk.IcechunkError:   x error writing object to object store service error
  |
  | context:
  |    0: icechunk::storage::s3::write_chunk
  |            with id=b903788d154ced89831aab01
  |              at icechunk/src/storage/s3.rs:442
  |    1: icechunk::asset_manager::write_chunk
  |            with chunk_id=b903788d154ced89831aab01
  |              at icechunk/src/asset_manager.rs:259
  |    2: icechunk::store::set
  |            with key="1x721x1440/10m_u_component_of_wind/c/164740/0/0"
  |              at icechunk/src/store.rs:276
  |
  |-> error writing object to object store service error
  |-> service error
  |-> unhandled error
  `-> error parsing XML: no root element

[dcherian]

Now I get

icechunk.IcechunkError:   x error writing object to object store dispatch failure
  |
  | context:
  |    0: icechunk::storage::s3::write_chunk
  |            with id=7d2113037e0cf584d3573729
  |              at icechunk/src/storage/s3.rs:442
  |    1: icechunk::asset_manager::write_chunk
  |            with chunk_id=7d2113037e0cf584d3573729
  |              at icechunk/src/asset_manager.rs:259
  |    2: icechunk::store::set
  |            with key="1x721x1440/10m_v_component_of_wind/c/336173/0/0"
  |              at icechunk/src/store.rs:276
  |
  |-> error writing object to object store dispatch failure
  |-> dispatch failure
  |-> io error
  |-> connection error: Connection reset by peer (os error 104)
  `-> Connection reset by peer (os error 104)

which seems like network flakiness, do we have retries for this yet? This job is running in AWS using Coiled btw, so it's not my home network.