Feat/s3 delete files

[lbeckman314]

New Features

• Adds support for deleting files from non-AWS S3-compatible endpoints (e.g. MinIO, Wasabi, Ceph).

Breaking Changes

• None

Bug Fixes

• Fixes the "The AWS Access Key Id you provided does not exist" error when attempting to delete a file from a non-AWS bucket.

Improvements

• Allows for Gen3 administrators to have greater flexibility in choosing their preferred data storage, while keeping the AWS S3 endpoint as the "default" storage option.

Dependency updates

• None

Deployment changes

• None

Initial Bug Report

Issue

When attempting to delete an indexd record using the delete_file_locations() method in the Gen3 SDK, we encounter a “The AWS Access Key Id you provided does not exist” error. The indexd record is in non-AWS S3 bucket (MinIO endpoint), which is specified by the endpoint_url of the bucket in the Fence config.
 

Expected Behavior

The delete_file_locations() method should delete both the file and the indexd record for both AWS and non-AWS S3-compatible buckets.

Error Message

[2023-11-17 21:33:45,499][fence.blueprints.data.blueprint][   INFO] Deleting record and files for <EXAMPLE>

[2023-11-17 21:33:45,504][fence.blueprints.data.indexd][   INFO] Attempting to delete file named <EXAMPLE> from bucket example-production.

[2023-11-17 21:33:45,803][fence.blueprints.data.indexd][  ERROR] An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.  <--- AWS Access Key Error

 

Possible Cause

It might be possible the the endpoint_url of the bucket isn’t being passed to the Boto client in the Fence code?
 
fence-config.yaml:

      example-bucket:
        cred: 'EXAMPLE-USER'
        programs:
        - example_program
        endpoint_url: https://minio-storage.example.edu/  <-- Endpoint URL of the MinIO S3 Bucket

[lbeckman314]

Working on running all unit tests associated with this PR.

[lbeckman314]

Hi @BinamB Thank you for your patience with this PR on non-AWS S3 file deletion. Unit tests are now all passing, and we are finishing up the documentation of our integration test (tested against AWS and MinIO Buckets).

• Fence PR: Feat/s3 delete files #1127
• Travis CI (Unit Tests): https://app.travis-ci.com/github/uc-cdis/fence/builds/268544693
• Target PR/Branch: (PPS-411): Add bucket selection support for multi part upload #1112
• Blocking?: Negative — we are able to use the forked branch in our deployments.

Not sure what’s blocking the Jenkins CI run, happy to take a look at any potential issues there.

Let us know if we can provide anything to help with any reviews, thanks again!

[BinamB]

Everything looks good to me. Could you add a unit test for delete_data_file such that it can test the new get_s3_client and a separate test for create_s3_clients? @lbeckman314