aws-s3-sdk 1.20.0 hangs before being able to create a session and connect

[crisidev]

Describe the bug

We upgraded aws-s3-sdk from 1.15.0 to 1.20.0 and discovered it silently hangs before even being able to try to connect to specified endpoint.

NOTE: we are running against a self hosted minio S3 and thought this was an issue with our infrastructure setup, but tcpdump shows no connection attempts

Reverting to 1.15.0 solves the problem for us.

Expected Behavior

The expected behaviour would be for the program to finish and to see an attempt to connect to localhost:9000 with tcpdump.

Since the endpoint is "fake", no real download operation is expected.

Current Behavior

The program hangs at start download (see Reproduction Steps) and an attempt to connect should be printed by tcpdump:

12:22:41.781751 lo    In  IP 127.0.0.1.56286 > 127.0.0.1.9000: tcp 0
12:22:41.781761 lo    In  IP 127.0.0.1.9000 > 127.0.0.1.56286: tcp 0

Reproduction Steps

We managed to reproduce the issue by using this simple main.rs. Please note you don't need a running endpoint on https://127.0.0.1:9000 to reproduce this.

use aws_sdk_s3::config::endpoint::Endpoint;
use aws_sdk_s3::config::endpoint::EndpointFuture;
use aws_sdk_s3::config::endpoint::Params;
use aws_sdk_s3::config::endpoint::ResolveEndpoint;
use aws_sdk_s3::config::Region;
use tracing_subscriber::fmt;
use tracing_subscriber::EnvFilter;
use tracing_subscriber::prelude::*;

#[tokio::main]
async fn main() {
    let _ = tracing_subscriber::registry()
            .with(fmt::layer().with_ansi(false))
            .with(EnvFilter::from_default_env())
            .init();

    let access_key_id = "anaccesskey".to_string();
    let secret_access_key = "asecretkey".to_string();
    let endpoint_url = "https://127.0.0.1:9000".to_string();
    let bucket_name = "abucket".to_string();

    println!("init credentials");
    let credentials = aws_credential_types::Credentials::from_keys(
        access_key_id,
        secret_access_key,
        None,
    );

    println!("init config");
    let config = aws_sdk_s3::Config::builder()
        .region(Region::new("dummy"))
        .credentials_provider(credentials)
        .endpoint_resolver(S3EndpointResolver {
            url: endpoint_url,
        })
        .build();

    println!("init client");
    let client = aws_sdk_s3::Client::from_conf(config);
    println!("start download");
    let _ = client
            .get_object()
            .bucket(bucket_name)
            .key(format!("test"))
            .send()
            .await;

    println!("done download");
}

#[derive(Debug)]
struct S3EndpointResolver {
    url: String,
}
impl ResolveEndpoint for S3EndpointResolver {
    fn resolve_endpoint(&self, params: &Params) -> EndpointFuture<'_> {
        let full_url = format!("{}/{}", self.url, params.bucket().unwrap_or(""));
        EndpointFuture::ready(Ok(Endpoint::builder().url(full_url).build()))
    }
}

with this Cargo.toml

[package]
name = "brokens3"
version = "0.1.0"
edition = "2021"

[dependencies]
aws-config = { version = "=1.1.8", features = ["behavior-version-latest"] }
aws-credential-types = { version = "=1.1.8", features = ["hardcoded-credentials"] }
aws-sdk-s3 = { version = "=1.20.0", features = ["behavior-version-latest"] }
aws-smithy-runtime-api = "=1.2.0"
tokio = { version = "1.36.0", features = ["full"] }
tracing-subscriber = { version = "0.3.18", features = ["env-filter"] }

This code just hangs at "start download":

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.22s
     Running `target/debug/brokens3`
init credentials
init config
init client
start download

Possible Solution

No response

Additional Information/Context

No response

Version

## Affected versions:
├── aws-config v1.1.8
│   ├── aws-credential-types v1.1.8
│   │   ├── aws-smithy-async v1.1.8
│   │   ├── aws-smithy-runtime-api v1.2.0
│   │   │   ├── aws-smithy-async v1.1.8 (*)
│   │   │   ├── aws-smithy-types v1.1.8
│   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-runtime v1.1.8
│   │   ├── aws-credential-types v1.1.8 (*)
│   │   ├── aws-sigv4 v1.2.0
│   │   │   ├── aws-credential-types v1.1.8 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   │   ├── aws-smithy-http v0.60.7
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.7 (*)
│   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.1.8
│   │   │   ├── aws-credential-types v1.1.8 (*)
│   │   │   ├── aws-smithy-async v1.1.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-sdk-sso v1.17.0
│   │   ├── aws-credential-types v1.1.8 (*)
│   │   ├── aws-runtime v1.1.8 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-http v0.60.7 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime v1.1.8
│   │   │   ├── aws-smithy-async v1.1.8 (*)
│   │   │   ├── aws-smithy-http v0.60.7 (*)
│   │   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.1.8 (*)
│   ├── aws-sdk-ssooidc v1.17.0
│   │   ├── aws-credential-types v1.1.8 (*)
│   │   ├── aws-runtime v1.1.8 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-http v0.60.7 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.1.8 (*)
│   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.1.8 (*)
│   ├── aws-sdk-sts v1.17.0
│   │   ├── aws-credential-types v1.1.8 (*)
│   │   ├── aws-runtime v1.1.8 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-http v0.60.7 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime v1.1.8 (*)
│   │   ├── aws-smithy-runtime-api v1.2.0 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-xml v0.60.7
│   │   ├── aws-types v1.1.8 (*)
│   ├── aws-smithy-async v1.1.8 (*)
│   ├── aws-smithy-http v0.60.7 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.1.8 (*)
│   ├── aws-smithy-runtime-api v1.2.0 (*)
│   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-types v1.1.8 (*)
├── aws-credential-types v1.1.8 (*)
├── aws-sdk-s3 v1.20.0
│   ├── aws-credential-types v1.1.8 (*)
│   ├── aws-runtime v1.1.8 (*)
│   ├── aws-sigv4 v1.2.0 (*)
│   ├── aws-smithy-async v1.1.8 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.7 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.7 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.1.8 (*)
│   ├── aws-smithy-runtime-api v1.2.0 (*)
│   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-smithy-xml v0.60.7 (*)
│   ├── aws-types v1.1.8 (*)
├── aws-smithy-runtime-api v1.2.0 (*)

## Working versions
├── aws-config v1.1.7
│   ├── aws-credential-types v1.1.7
│   │   ├── aws-smithy-async v1.1.8
│   │   ├── aws-smithy-runtime-api v1.1.7
│   │   │   ├── aws-smithy-async v1.1.8 (*)
│   │   │   ├── aws-smithy-types v1.1.8
│   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-runtime v1.1.7
│   │   ├── aws-credential-types v1.1.7 (*)
│   │   ├── aws-sigv4 v1.1.7
│   │   │   ├── aws-credential-types v1.1.7 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   │   ├── aws-smithy-http v0.60.6
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.6 (*)
│   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.1.7
│   │   │   ├── aws-credential-types v1.1.7 (*)
│   │   │   ├── aws-smithy-async v1.1.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-sdk-sso v1.15.0
│   │   ├── aws-credential-types v1.1.7 (*)
│   │   ├── aws-runtime v1.1.7 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-http v0.60.6 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime v1.1.7
│   │   │   ├── aws-smithy-async v1.1.8 (*)
│   │   │   ├── aws-smithy-http v0.60.6 (*)
│   │   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.1.7 (*)
│   ├── aws-sdk-ssooidc v1.15.0
│   │   ├── aws-credential-types v1.1.7 (*)
│   │   ├── aws-runtime v1.1.7 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-http v0.60.6 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.1.7 (*)
│   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-types v1.1.7 (*)
│   ├── aws-sdk-sts v1.15.0
│   │   ├── aws-credential-types v1.1.7 (*)
│   │   ├── aws-runtime v1.1.7 (*)
│   │   ├── aws-smithy-async v1.1.8 (*)
│   │   ├── aws-smithy-http v0.60.6 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-runtime v1.1.7 (*)
│   │   ├── aws-smithy-runtime-api v1.1.7 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   │   ├── aws-smithy-xml v0.60.7
│   │   ├── aws-types v1.1.7 (*)
│   ├── aws-smithy-async v1.1.8 (*)
│   ├── aws-smithy-http v0.60.6 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.1.7 (*)
│   ├── aws-smithy-runtime-api v1.1.7 (*)
│   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-types v1.1.7 (*)
├── aws-credential-types v1.1.7 (*)
├── aws-sdk-s3 v1.17.0
│   ├── aws-credential-types v1.1.7 (*)
│   ├── aws-runtime v1.1.7 (*)
│   ├── aws-sigv4 v1.1.7 (*)
│   ├── aws-smithy-async v1.1.8 (*)
│   ├── aws-smithy-checksums v0.60.6
│   │   ├── aws-smithy-http v0.60.6 (*)
│   │   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.6 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.1.7 (*)
│   ├── aws-smithy-runtime-api v1.1.7 (*)
│   ├── aws-smithy-types v1.1.8 (*)
│   ├── aws-smithy-xml v0.60.7 (*)
│   ├── aws-types v1.1.7 (*)
├── aws-smithy-runtime-api v1.1.7 (*)

Environment details (OS name and version, etc.)

Macos and Linux

Logs

No response

[rcoh]

I believe this was a bug in our implementation of S3 Express. A fix should be released this week. The issue comes because S3Express inadvertently come before Sigv4 in the signing implementation.

To fix it, use .endpoint_url instead of .endpoint_resolver. That will ensure that the underlying auth scheme is respected. When you fully replace the endpoint_resolver, you're also removing the chosen authentication scheme.

[crisidev]

Thanks Russell!! I'll give it a shot and report back. Nice to be back bothering you 😛

[crisidev]

I confirm using .endpoint_url fixes the issue. Thanks a lot.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[ysaito1001]

@crisidev 👋 The fix was released today. While we should use .endpoint_url, the fix ensures that it should not hang even if you use .endpoint_resolver.