htsget download endpoints

[MalinAhlberg]

Related issue(s) and PR(s)
This PR closes NBISweden/LocalEGA-SE-Deployment#725 and neicnordic/sda-download#372.

It is based on the work in neicnordic/sda-download#327.

Description

• Adds endpoint /s3-encrypted which is similar to /s3 but returns an encrypted file.
• Adds support for returning parts of encrypted files, either with url param endCoordinate or - for encrypted files - with the http header Range: bytes=0-1000. The later overrides the first, and is strictly interpreted, while endCoordinate will be used to find the nearest crypt4gh data block boundary and cut the file there.
• Minor modifications to work with htsget-rs (see https://github.com/GenomicDataInfrastructure/starter-kit-htsget/tree/feature/rust-htsget)

How to test

• get a token: token=$(curl -s -k https://localhost:8080/tokens | jq -r '.[0]')
• to download the complete file: curl 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam' -H "Authorization: Bearer $token" -o test.bam.c4gh
• to download the first 3000 bytes: curl 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam' -H "Authorization: Bearer $token" -o test3000.bam.c4gh -H "Range: bytes=0-3000
• to download the first 3000 bytes, and adding extra bytes to align to the data package boundaries : curl 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam?endCoordinate=3000' -H "Authorization: Bearer $token" -o test3000ok.bam.c4gh
• to get the file info curl --head -i 'http://localhost:8443/s3-encrypted/DATASET0001/htsnexus_test_NA12878.bam' -H "Authorization: Bearer $token"

Further comments

• startCoordinate is not implemented (see [sda-download] Implement random access in encrypted files #696)
• also see [sda-auth] Add htsget to s3config #400 and [sda-download] Connect re-encryption service #698 for future work that will be required

[blankdots]

small things

[pontus]

I'm not sure what's in scope for this PR, should header reencryption be included or not? Some things may be premature without that.

[MalinAhlberg]

I'm not sure what's in scope for this PR, should header reencryption be included or not? Some things may be premature without that.

Valid points! The scope of this PR is to be able to communicate and do requests with htsget + sda-download. Some future work is mentioned under Further comments in the PR body.
We will use these headers in communication with htsget, for the header encryption.

So, no, all things are not in place yet, but maybe good as a first step?

[blankdots]

I would say, if we can keep the changes in the scope of htsget download endpoints that is ok, but removing the c4gh.password and adding info and options about re-encryption e.g. Client-public-key or Server-public-key maybe we should do the changes with that, as it will be easier to trace.

the reason i am mentioning this is because is while for htsget re-encryption might work in a certain way and that is ok, for serving decrypted filles we might need to generate fresh key pair every time (see changes in sda.go from https://github.com/neicnordic/sda-download/pull/330/files) and those changes need to come together.

[jbygdell]

There are some tests and error handling that's missing

[aaperis]

Looks good!

Requesting new review