Bump software.amazon.awssdk from 2.20.86 to 2.30.18 #17396

peterzhuamazon · 2025-02-19T22:24:56Z

Description

Bump software.amazon.awssdk from 2.20.86 to 2.30.18

Related Issues

Follow up to #17320

Check List

~~[ ] Functionality includes testing.~~
~~[ ] API changes companion pull request created, if applicable.~~
~~[ ] Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

github-actions · 2025-02-19T22:36:04Z

❌ Gradle check result for 5e94114: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

peterzhuamazon · 2025-02-19T23:45:31Z

Still need sha / license / notice changes.

peterzhuamazon · 2025-02-19T23:47:38Z


> Task :plugins:discovery-ec2:thirdPartyAudit FAILED
Missing classes:

github-actions · 2025-02-19T23:58:33Z

❌ Gradle check result for 1a7a762: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

github-actions · 2025-02-20T20:07:58Z

❌ Gradle check result for daa5d56: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

peterzhuamazon · 2025-02-20T20:33:55Z

Not sure why it is not found:

> Task :plugins:crypto-kms:test

REPRODUCE WITH: ./gradlew ':plugins:crypto-kms:test' --tests "org.opensearch.crypto.kms.KmsServiceTests.testClientSettingsReInit" -Dtests.seed=ABC1ADE894899B55 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=America/Winnipeg -Druntime.java=23

KmsServiceTests > testClientSettingsReInit FAILED
    java.lang.NoClassDefFoundError: software/amazon/awssdk/retries/api/RetryStrategy
        at __randomizedtesting.SeedInfo.seed([ABC1ADE894899B55:1F1AC593463AB41C]:0)
        at software.amazon.awssdk.core.client.config.SdkClientOption.<clinit>(SdkClientOption.java:69)
        at software.amazon.awssdk.core.client.config.ClientOverrideConfiguration.<clinit>(ClientOverrideConfiguration.java:109)
        at org.opensearch.crypto.kms.KmsService.buildOverrideConfiguration(KmsService.java:136)
        at org.opensearch.crypto.kms.KmsService.buildClient(KmsService.java:75)
        at org.opensearch.crypto.kms.KmsService.lambda$client$2(KmsService.java:169)
        at java.****/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.opensearch.crypto.kms.SocketAccess.doPrivileged(SocketAccess.java:29)
        at org.opensearch.crypto.kms.KmsService.client(KmsService.java:169)
        at org.opensearch.crypto.kms.KmsServiceTests.testClientSettingsReInit(KmsServiceTests.java:119)

        Caused by:
        java.lang.ClassNotFoundException: software.amazon.awssdk.retries.api.RetryStrategy
            at java.****/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
            at java.****/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
            at java.****/java.lang.ClassLoader.loadClass(ClassLoader.java:528)
            ... 9 more

REPRODUCE WITH: ./gradlew ':plugins:crypto-kms:test' --tests "org.opensearch.crypto.kms.KmsServiceTests.testAWSConfigurationWithAwsSettings" -Dtests.seed=ABC1ADE894899B55 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=America/Winnipeg -Druntime.java=23

KmsServiceTests > testAWSConfigurationWithAwsSettings FAILED
    java.lang.NoClassDefFoundError: Could not initialize class software.amazon.awssdk.core.client.config.ClientOverrideConfiguration
        at __randomizedtesting.SeedInfo.seed([ABC1ADE894899B55:CC9E4A455F7D29B8]:0)
        at org.opensearch.crypto.kms.KmsService.buildOverrideConfiguration(KmsService.java:136)
        at java.****/java.security.AccessController.doPrivileged(AccessController.java:319)
        at org.opensearch.crypto.kms.SocketAccess.doPrivileged(SocketAccess.java:29)
        at org.opensearch.crypto.kms.KmsServiceTests.testAWSConfigurationWithAwsSettings(KmsServiceTests.java:79)

        Caused by:
        java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: software/amazon/awssdk/retries/api/RetryStrategy [in thread "TEST-KmsServiceTests.testClientSettingsReInit-seed#[ABC1ADE894899B55]"]
            at software.amazon.awssdk.core.client.config.SdkClientOption.<clinit>(SdkClientOption.java:69)
            at software.amazon.awssdk.core.client.config.ClientOverrideConfiguration.<clinit>(ClientOverrideConfiguration.java:109)
            at org.opensearch.crypto.kms.KmsService.buildOverrideConfiguration(KmsService.java:136)
            at org.opensearch.crypto.kms.KmsService.buildClient(KmsService.java:75)
            at org.opensearch.crypto.kms.KmsService.lambda$client$2(KmsService.java:169)
            at java.****/java.security.AccessController.doPrivileged(AccessController.java:319)
            at org.opensearch.crypto.kms.SocketAccess.doPrivileged(SocketAccess.java:29)
            at org.opensearch.crypto.kms.KmsService.client(KmsService.java:169)
            at org.opensearch.crypto.kms.KmsServiceTests.testClientSettingsReInit(KmsServiceTests.java:119)

REPRODUCE WITH: ./gradlew ':plugins:crypto-kms:test' --tests "org.opensearch.crypto.kms.KmsServiceTests.testAWSDefaultConfiguration" -Dtests.seed=ABC1ADE894899B55 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ms-MY -Dtests.timezone=America/Winnipeg -Druntime.java=23

KmsServiceTests > testAWSDefaultConfiguration FAILED
    java.lang.AssertionError: expected null, but was:<http>
        at __randomizedtesting.SeedInfo.seed([ABC1ADE894899B55:194FC9BE64D23FFB]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotNull(Assert.java:756)
        at org.junit.Assert.assertNull(Assert.java:738)
        at org.junit.Assert.assertNull(Assert.java:748)
        at org.opensearch.crypto.kms.KmsServiceTests.testAWSDefaultConfiguration(KmsServiceTests.java:34)

cwperks · 2025-02-20T20:39:02Z

plugins/crypto-kms/build.gradle

@@ -30,6 +30,7 @@ dependencies {
  api "software.amazon.awssdk:aws-core:${versions.aws}"
  api "software.amazon.awssdk:utils:${versions.aws}"
  api "software.amazon.awssdk:auth:${versions.aws}"
+  api "software.amazon.awssdk:identity-spi:${versions.aws}"


Does there need to be a dependency on https://mvnrepository.com/artifact/software.amazon.awssdk/retries-spi as well?

Interesting, crypto-kms didnt show any thirdparty missing classes.
let me check.

Signed-off-by: Andrew Ross <andrross@amazon.com>

andrross · 2025-02-20T20:42:40Z

@peterzhuamazon @cwperks I just pushed a commit to fix the crypto-kms failures

plugins/crypto-kms/src/test/java/org/opensearch/crypto/kms/KmsServiceTests.java

Signed-off-by: Andrew Ross <andrross@amazon.com>

github-actions · 2025-02-20T21:03:22Z

❌ Gradle check result for 60ae79e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andrew Ross <andrross@amazon.com>

github-actions · 2025-02-20T21:25:55Z

❌ Gradle check result for 058fe8b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Andrew Ross <andrross@amazon.com>

github-actions · 2025-02-20T22:21:14Z

❌ Gradle check result for e6e5066: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

peterzhuamazon · 2025-02-20T22:42:01Z

@andrross seems like more failures on repository-s3 test:

> Task :plugins:repository-s3:test
AwsS3ServiceImplTests > testIrsaCredentialsFromKeystore FAILED
    java.security.AccessControlException: access denied ("java.io.FilePermission" "config" "read")
        at __randomizedtesting.SeedInfo.seed([7313EA5FD344A1A5:A26BB6BBB13F40DF]:0)
        at java.****/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)
        at java.****/java.security.AccessController.checkPermission(AccessController.java:1085)
        at java.****/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)
        at java.****/java.lang.SecurityManager.checkRead(SecurityManager.java:742)
        at java.****/sun.nio.fs.UnixPath.checkRead(UnixPath.java:840)
        at java.****/sun.nio.fs.UnixFileSystemProvider.readAttributesIfExists(UnixFileSystemProvider.java:183)
        at java.****/java.nio.file.Files.isRegularFile(Files.java:2370)
        at software.amazon.awssdk.profiles.ProfileFileLocation.lambda$resolveIfExists$2(ProfileFileLocation.java:90)
        at java.****/java.util.Optional.filter(Optional.java:218)
        at software.amazon.awssdk.profiles.ProfileFileLocation.resolveIfExists(ProfileFileLocation.java:90)
        at software.amazon.awssdk.profiles.ProfileFileLocation.credentialsFileLocation(ProfileFileLocation.java:77)
        at software.amazon.awssdk.profiles.ProfileFile.addCredentialsFile(ProfileFile.java:151)
        at software.amazon.awssdk.utils.builder.SdkBuilder.applyMutation(SdkBuilder.java:61)
        at software.amazon.awssdk.profiles.ProfileFile.defaultProfileFile(ProfileFile.java:99)
        at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.lambda$new$0(InstanceProfileCredentialsProvider.java:102)
        at java.****/java.util.Optional.orElseGet(Optional.java:364)
        at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:102)
        at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.<init>(InstanceProfileCredentialsProvider.java:66)
        at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider$BuilderImpl.build(InstanceProfileCredentialsProvider.java:468)
        at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.initializeProvider(S3Service.java:471)
        at org.opensearch.repositories.s3.S3Service$PrivilegedInstanceProfileCredentialsProvider.<init>(S3Service.java:461)
        at org.opensearch.repositories.s3.S3Service.buildCredentials(S3Service.java:419)
        at org.opensearch.repositories.s3.AwsS3ServiceImplTests.testIrsaCredentialsFromKeystore(AwsS3ServiceImplTests.java:203)

peterzhuamazon · 2025-02-20T22:59:02Z


AwsS3ServiceImplTests > testIrsaCredentialsFromKeystore FAILED
    software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@53d570b9: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@2e8d7f8e: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@439b921d: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@2ae11c98: EC2 Metadata is disabled. Unable to retrieve region information from EC2 Metadata service.]

I think we already have this set in gradle check env.

peterzhuamazon · 2025-02-20T23:43:35Z

Hi @reta @andrross @cwperks is this related to the JSM?

    java.security.AccessControlException: access denied ("java.io.FilePermission" "config" "read")

Thanks.

reta · 2025-02-20T23:48:53Z

Hi @reta @andrross @cwperks is this related to the JSM?
    java.security.AccessControlException: access denied ("java.io.FilePermission" "config" "read")
Thanks.

Yes! So we do not allow FS access outside of OPENSEARCH_CONF folder, seems like AWS SDK wants to read config file

andrross · 2025-02-20T23:54:28Z

FYI @vikasvb90 @sachinpkale @ashking94

It looks like there are some significant changes in the AWS SDK for this minor version update. I know you folks have worked in this area in the past so I'd appreciate it if you could take a look here. Thanks!

peterzhuamazon · 2025-02-20T23:55:46Z

Hi @reta @andrross @cwperks is this related to the JSM?
    java.security.AccessControlException: access denied ("java.io.FilePermission" "config" "read")
Thanks.
Yes! So we do not allow FS access outside of OPENSEARCH_CONF folder, seems like AWS SDK wants to read config file

What would be a good approach right now?

Do not update aws sdk version? And hardcode netty version as well in ml?
Find a way to update the test?

Thanks.

reta · 2025-02-21T00:03:05Z

What would be a good approach right now?\n\nDo not update aws sdk version? And hardcode netty version as well in ml?\nFind a way to update the test?\nThanks.

Thanks @peterzhuamazon , I think we could postpone AWS SDK update for core, but it is largely unrelated to ml-commons - it could use the latest AWS SDK, I think you updated the plugin already?

peterzhuamazon · 2025-02-21T00:14:24Z

What would be a good approach right now?\n\nDo not update aws sdk version? And hardcode netty version as well in ml?\nFind a way to update the test?\nThanks.

Thanks @peterzhuamazon , I think we could postpone AWS SDK update for core, but it is largely unrelated to ml-commons - it could use the latest AWS SDK, I think you updated the plugin already?

We were planning to update but paused a bit because we want to update core then source the version from core.

But now seems like we will just update plugins directly.

Thanks.

cwperks · 2025-02-21T02:43:13Z

Hi @reta @andrross @cwperks is this related to the JSM?
    java.security.AccessControlException: access denied ("java.io.FilePermission" "config" "read")
Thanks.
Yes! So we do not allow FS access outside of OPENSEARCH_CONF folder, seems like AWS SDK wants to read config file

Looks like there's an issue filed on the AWS SDK repo here: aws/aws-sdk-java-v2#5754 (comment)

Bump software.amazon.awssdk from 2.20.86 to 2.30.18

5e94114

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

peterzhuamazon added release backport 2.x Backport to 2.x branch backport 2.19 labels Feb 19, 2025

peterzhuamazon self-assigned this Feb 19, 2025

peterzhuamazon mentioned this pull request Feb 19, 2025

[RELEASE] Release version 2.19.1 opensearch-project/opensearch-build#5323

Open

64 tasks

peterzhuamazon added 3 commits February 20, 2025 19:35

Update license and missing classes on plugins

6e7efc2

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

Update version dynamically

d471dbe

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

Merge remote-tracking branch 'upstream/main' into bump-aws-sdk-2.30.18

daa5d56

cwperks reviewed Feb 20, 2025

View reviewed changes

Fix crypto-kms dependency and test issues

b07b983

Signed-off-by: Andrew Ross <andrross@amazon.com>

andrross reviewed Feb 20, 2025

View reviewed changes

plugins/crypto-kms/src/test/java/org/opensearch/crypto/kms/KmsServiceTests.java Show resolved Hide resolved

Fix unit tests in discovery-ec2

60ae79e

Signed-off-by: Andrew Ross <andrross@amazon.com>

Fix SHAs for crypto-kms

058fe8b

Signed-off-by: Andrew Ross <andrross@amazon.com>

Add license files

e6e5066

Signed-off-by: Andrew Ross <andrross@amazon.com>

peterzhuamazon mentioned this pull request Feb 20, 2025

address CVEs CVE-2025-25193, CVE-2025-24970, CVE-2024-57699 opensearch-project/ml-commons#3575

Merged

5 tasks

peterzhuamazon marked this pull request as draft February 21, 2025 00:15

andrross mentioned this pull request Feb 21, 2025

Inaccessible profile file (due to JVM security manager) blocks client creation aws/aws-sdk-java-v2#5754

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump software.amazon.awssdk from 2.20.86 to 2.30.18 #17396

Bump software.amazon.awssdk from 2.20.86 to 2.30.18 #17396

peterzhuamazon commented Feb 19, 2025

github-actions bot commented Feb 19, 2025

peterzhuamazon commented Feb 19, 2025

peterzhuamazon commented Feb 19, 2025

github-actions bot commented Feb 19, 2025

github-actions bot commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

cwperks Feb 20, 2025

peterzhuamazon Feb 20, 2025

andrross commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

reta commented Feb 20, 2025

andrross commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

reta commented Feb 21, 2025

peterzhuamazon commented Feb 21, 2025

cwperks commented Feb 21, 2025

Bump software.amazon.awssdk from 2.20.86 to 2.30.18 #17396

Are you sure you want to change the base?

Bump software.amazon.awssdk from 2.20.86 to 2.30.18 #17396

Conversation

peterzhuamazon commented Feb 19, 2025

Description

Related Issues

Check List

github-actions bot commented Feb 19, 2025

peterzhuamazon commented Feb 19, 2025

peterzhuamazon commented Feb 19, 2025

github-actions bot commented Feb 19, 2025

github-actions bot commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

cwperks Feb 20, 2025

Choose a reason for hiding this comment

peterzhuamazon Feb 20, 2025

Choose a reason for hiding this comment

andrross commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

github-actions bot commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

reta commented Feb 20, 2025

andrross commented Feb 20, 2025

peterzhuamazon commented Feb 20, 2025

reta commented Feb 21, 2025

peterzhuamazon commented Feb 21, 2025

cwperks commented Feb 21, 2025