Feature/s3

[sandorkertesz]

Fixes #223

This PR adds the "s3" source to access Amazon S3 buckets.

import earthkit.data

req = {
    "endpoint": "object-store.os-api.cci1.ecmwf.int",
    "bucket": "earthkit-test-data-public",
    "objects": "test6.grib",
}

ds = earthkit.data.from_source("s3", req, anon=True)

More examples are available at: https://earthkit-data.readthedocs.io/en/feature-s3/examples/s3.html

Documentation

https://earthkit-data.readthedocs.io/en/feature-s3/guide/sources.html#s3

Highlights

• Multiple buckets and objects can be used
• Part(s) of an object can be specified
• Can be read as a stream. By default stream=False
• The anon option controls whether if it is an anonymous access or AWS credentials should be used. The default is anon=True.
• New optional dependencies:
  • aws-requests-auth: Only used when anon=False.
  • botocore. Only used when anon=False and credentials are not specified via the kwargs.

optional-dependencies.s3 = [
  "aws-requests-auth",
  "botocore",
]

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 74.25150% with 43 lines in your changes missing coverage. Please review.

Project coverage is 90.68%. Comparing base (e7bfd19) to head (518dc59).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
tests/sources/test_s3.py	64.64%	33 Missing and 2 partials ⚠️
tests/utils/test_url.py	88.46%	2 Missing and 1 partial ⚠️
tests/utils/test_utils.py	75.00%	2 Missing and 1 partial ⚠️
tests/sources/test_url.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #484      +/-   ##
===========================================
- Coverage    90.98%   90.68%   -0.30%     
===========================================
  Files          139      142       +3     
  Lines         9581     9739     +158     
  Branches       466      459       -7     
===========================================
+ Hits          8717     8832     +115     
- Misses         712      751      +39     
- Partials       152      156       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.