retry token bucket not set

[aajtodd]

Describe the bug

While investigating a cross SDK issue I discovered we are not setting a token bucket in the config bag. This causes retries to only fail when max attempts are exceeded not when the token bucket has exhausted it's retry quota.

This appears to have been unintentionally removed in smithy-lang/smithy-rs#3071.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

We should fail long before a large number of max attempts are hit if every request is failing as the retry quota should be exhausted

Current Behavior

Failure only happens when max attempts are exceeded.

Without a token bucket configured the standard retry strategy never acquires permits.

Reproduction Steps

Reproduction, logs show we only ever fail on max retries.

use std::{error::Error, sync::{atomic::{AtomicBool, Ordering}, Arc}, time::Duration};
use aws_config::retry::RetryConfig;
use aws_smithy_runtime::client::http::test_util::infallible_client_fn;
use aws_sdk_sts::error::DisplayErrorContext;

type BoxError = Box<dyn Error + Send + Sync>;

#[tokio::main]
async fn main() -> Result<(), BoxError> {
    tracing_subscriber::fmt::init();

    let error_xml = r#"<ErrorResponse>
    <Error>
        <Type>Server</Type>
        <Code>Unknown Error</Code>
        <Message>Test bucket behavior</Message>
    </Error>
    <RequestId>foo-id</RequestId>
</ErrorResponse>"#;

    let success_xml = r#"
<GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <GetCallerIdentityResult>
  <Arn>arn:aws:iam::123456789012:user/Alice</Arn>
  <UserId>AIDACKCEVSQ6C2EXAMPLE</UserId>
  <Account>123456789012</Account>
  </GetCallerIdentityResult>
  <ResponseMetadata>
    <RequestId>01234567-89ab-cdef-0123-456789abcdef</RequestId>
  </ResponseMetadata>
</GetCallerIdentityResponse>"#;

    let allow_success = Arc::new(AtomicBool::new(false));

    let allowed = allow_success.clone();
    let http_client = infallible_client_fn(move |_req| {
        let allow_success = allowed.load(Ordering::SeqCst);
        if allow_success {
            http_02x::Response::builder().status(200).body(success_xml).unwrap()
        }else {
            http_02x::Response::builder().status(500).body(error_xml).unwrap()
        }
    });
    
    let aws_config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .retry_config(
            RetryConfig::standard().with_max_attempts(100).with_max_backoff(Duration::from_millis(10))
        )
        .http_client(http_client)
        .load().await;

    let sts = aws_sdk_sts::Client::new(&aws_config);
    let result = sts.get_caller_identity().send().await;
    let err = result.expect_err("error expected");
    println!("error: {:#?}", DisplayErrorContext(&err));

    println!("second attempt!!!");

    allow_success.store(true, Ordering::SeqCst);
    let result2 = sts.get_caller_identity().send().await;
    let _ = result2.expect("expect success");

    println!("third attempt!!!");

    allow_success.store(false, Ordering::SeqCst);
    let result3 = sts.get_caller_identity().send().await;
    let err3 = result3.expect_err("error expected");
    println!("error 3: {:#?}", DisplayErrorContext(&err3));


    println!("next client, first attempt!");
    let sts_2 = aws_sdk_sts::Client::new(&aws_config);
    let result4 = sts_2.get_caller_identity().send().await;
    let err4 = result4.expect_err("error expected");
    println!("error: {:#?}", DisplayErrorContext(&err4));
    
    Ok(())
}

Possible Solution

Token bucket should always be present. We probably want it tied to the retry partition at the very least although our default retry partitioning seems to only use the service name and doesn't account for e.g. region

Additional Information/Context

No response

Version

Tested with `aws-sdk-sts@1.54.0`, `aws-smithy-runtime@1.76.0`

Environment details (OS name and version, etc.)

macos

Logs

No response

[aajtodd]

Fix will go out in next smithy-rs release.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.