Recent boto3 versions break synapse-s3-storage-provider with non-AWS S3 providers

[timothyb89]

Describe the bug

Sorry to create noise here as this is explicitly a bug in a dependency, but I didn't see an existing discussion and imagine others might run into this until it's resolved upstream.

If using synapse-s3-storage-provider, most non-AWS S3-compatible storage providers are currently broken following a breaking change in boto3. If you upgrade, the new version with the breaking change will be pulled in and break media access when using e.g. Minio, Backblaze B2, etc.

See these related issues:

• synapse-s3-storage-provider cant connect to minio  matrix-org/synapse-s3-storage-provider#121
• Announcement: S3 default integrity change boto/boto3#4392

You'll see errors logged by synapse like the following:

Jan 19 07:44:41 matrix matrix-synapse[2825791]: 2025-01-19 07:44:41,550 - synapse.http.server - 146 - ERROR - GET-29 - Failed handle request via 'ThumbnailResource': <XForwardedForRequest at 0x7fba38053a40 method='GET' uri='/_matrix/client/v1/media/thumbnail/example.com/foo?width=40&height=40&method=crop&allow_redirect=true' clientproto='HTTP/1.1' site='8008'>
Jan 19 07:44:41 matrix matrix-synapse[2825791]: Traceback (most recent call last):
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/http/server.py", line 332, in _async_render_wrapper
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     callback_return = await self._async_render(request)
Jan 19 07:44:41 matrix matrix-synapse[2825791]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/http/server.py", line 544, in _async_render
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     callback_return = await raw_callback_return
Jan 19 07:44:41 matrix matrix-synapse[2825791]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/rest/client/media.py", line 168, in on_GET
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     await self.thumbnailer.respond_local_thumbnail(
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/media/thumbnailer.py", line 298, in respond_local_thumbnail
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     await self._select_and_respond_with_thumbnail(
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/media/thumbnailer.py", line 593, in _select_and_respond_with_thumbnail
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     responder = await self.media_storage.fetch_media(file_info)
Jan 19 07:44:41 matrix matrix-synapse[2825791]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/media/media_storage.py", line 217, in fetch_media
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     res: Any = await provider.fetch(path, file_info)
Jan 19 07:44:41 matrix matrix-synapse[2825791]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 922, in _wrapper
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     return await func(*args, **kwargs)
Jan 19 07:44:41 matrix matrix-synapse[2825791]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/synapse/media/storage_provider.py", line 135, in fetch
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     return await maybe_awaitable(self.backend.fetch(path, file_info))
Jan 19 07:44:41 matrix matrix-synapse[2825791]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/s3_storage_provider.py", line 218, in s3_download_task
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     resp = s3_client.get_object(Bucket=bucket, Key=key)
Jan 19 07:44:41 matrix matrix-synapse[2825791]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 569, in _api_call
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     return self._make_api_call(operation_name, kwargs)
Jan 19 07:44:41 matrix matrix-synapse[2825791]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 19 07:44:41 matrix matrix-synapse[2825791]:   File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 1023, in _make_api_call
Jan 19 07:44:41 matrix matrix-synapse[2825791]:     raise error_class(parsed_response, operation_name)
Jan 19 07:44:41 matrix matrix-synapse[2825791]: botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the GetObject operation: Unsupported header 'x-amz-checksum-mode' received for this API call.

Existing thumbnails and images/other media will also fail to load.

To Reproduce

Enable S3 storage to a non-AWS provider as explained in this document: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-s3.md, for example Backblaze B2.

matrix_synapse_ext_synapse_s3_storage_provider_enabled: true
# ...

Expected behavior
S3 access should work with 3rd party hosts.

Matrix Server:

• OS: Ubuntu Server 22.04
• Architecture amd64

Additional context

As a workaround, you can edit roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2 to pin an older boto3 release. Find the RUN pip install synapse-s3-storage-provider==... line, and insert 'boto3<1.36.0' 'botocore<1.36.0' just after pip install. See this git diff:

$  git diff
diff --git a/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2 b/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2
index 39367fe09..11abe7e6e 100644
--- a/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2
+++ b/roles/custom/matrix-synapse/templates/synapse/customizations/Dockerfile.j2
@@ -6,7 +6,7 @@ RUN pip install synapse-auto-accept-invite=={{ matrix_synapse_ext_synapse_auto_a
 {% endif %}

 {% if matrix_synapse_container_image_customizations_s3_storage_provider_installation_enabled %}
-RUN pip install synapse-s3-storage-provider=={{ matrix_synapse_ext_synapse_s3_storage_provider_version }}
+RUN pip install 'boto3<1.36.0' 'botocore<1.36.0' synapse-s3-storage-provider=={{ matrix_synapse_ext_synapse_s3_storage_provider_version }}
 {% endif %}

 {% if matrix_synapse_container_image_customizations_templates_enabled %}

Rerun just install-all and media should start loading again.

[spantaleev]

@aine-etke and I were just investigating this same problem. Thanks for providing a working/tested workaround.

People (like me) who were lucky enough to use the playbook before boto3/botocore v1.36.0 were released did not happen to hit the problem. People who updated their system more recently (with boto3/botocore v1.36.0+ being out and being buggy) did.

---

I've applied your suggested fix automatically for all new installations and have confirmed that it indeed installs older versions of boto3 and botocore, as expected:

# docker run --rm --entrypoint=/bin/sh localhost/matrixdotorg/synapse:v1.122.0-customized -c 'pip list | grep boto'

boto3                       1.35.99
botocore                    1.35.99

---

The workaround (which pins these dependencies to pre-v1.36.0 versions) can be turned off by setting matrix_synapse_container_image_customizations_s3_storage_provider_installation_old_boto_workaround_enabled to false.

At some point in the future, we can remove this workaround.