Feature requests: createBucket with EC replication type for s3 clients and the precedence order changes

[Lucas12138]

1. createBucket with EC replication type for s3 clients
   When creating a bucket with s3 client, we cannot specify the replication type (while the ozone client supports this).
   Can we add the support for the s3 client for this as well? Potentially, the input can be in the request headers.

2. the precedence order changes
   Currently, we found the precedence order (scenarios of the actual replication type used as follows:)

1> when there's no cluster level default replication setting and no client level storage class, the replication config will be null
2> with cluster level default replication setting as EC and no client level storage class, the replication config will be Ratis
3> with cluster level default replication setting as Ratis and no client level storage class, the replication config will be Ratis

The scenario 1 and 3 makes senses but for 2, we think it should fall back to the cluster level replication setting EC.

It looks like it can be achieved by simply removing these lines:

ozone/hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java

Lines 805 to 808 in f3a0dbd

	if (StringUtils.isEmpty(storageType)) {
	S3StorageType defaultStorageType = S3StorageType.getDefault(ozoneConfiguration);
	storageType = (defaultStorageType != null ? defaultStorageType.toString() : null);
	}

Can you help verify if it's the case? Thank you

[kerneltime]

cc @SaketaChalamchala @errose28

[SaketaChalamchala]

@Lucas12138 The above code snippet ensures that in the absence of "x-amz-storage-class" header the replication type is always RATIS and
even if "x-amz-storage-class" header we only support Ratis storage type atm. Removing this snippet alone might not be the only thing that's required here.

From what I can see, there are multiple things to be done here.

1. There is a known issue with setting client/server side default replication as EC. HDDS-10336
   This needs to be resolved.
2. S3StorageClass should support EC replication type. The default replication type should be client side replication config and fallback to server side replication config.

BucketEndpoint
3. Should use Storage class header "x-amz-storage-class" if available when creating the bucket.
Order of precedence should be:
"x-amz-storage-class"
client side replication config
server side replication config

ObjectEndpoint
4. Should use Storage class header "x-amz-storage-class" if available when creating the bucket.
Order of precedence should be:
"x-amz-storage-class"
bucket's replcation config
client side replication config
server side replication config

I created an a Epic to track all the required changes. HDDS-10980