Replication Type RATIS, STAND_ALONE, CHAINED and EC.

[mosleyit]

We have deployed ozone in a 3 node HA Cluster.

We have found that if a node goes down then writes to the s3 gateway will fail. Our assumption is that this is due to the strong consistency requirement of Ratis.

Our question is if we change

  <property>
    <name>ozone.server.default.replication.type</name>
    <value>RATIS</value>
    <tag>OZONE</tag>
    <description>
      Default replication type to be used while writing key into ozone. The
      value can be specified when writing the key, default is used when
      nothing is specified when creating key or no default value set at bucket.
      Supported values: RATIS, STAND_ALONE, CHAINED and EC.
    </description>
  </property>

Changing this to CHAINED does this just change to eventual consistency?
Are there any other settings where we can set the minimum replication prior to a write confirmation?
Are there any drawbacks to this?
Will it perform as expected in that if a node goes offline or two we will continue to be able to write and read?

Since we only have three nodes we do not want the cluster to effectively fall down if we loose a node.

[adoroszlai]

• Please don't use CHAINED, it was never implemented. OM will fail to start with that setting. The reference in config description is unfortunately left over (HDDS-12144). Thanks for pointing this out.
• Putting metadata and data on the same nodes is not recommended.
• Write may succeed to two datanodes if the third datanode goes down while writing the last block. Further writes will not work (Unable to allocate a container ... Unable to find enough nodes ... Required 3. Found 2.).
• Read works as long as you have at least one replica (with 3 datanodes: at least one of the three) available.

[mosleyit]

Okay thank you for that. We are currently running RATIS with OM and SCM running in HA.

We have three nodes. Each of the nodes is running OM SCM and DATANODE.

We have found that when 1 node goes down we can no longer PUT or GET files. We can perform list functions.

Is this the expected behavior?

If this is then the documentation should be updated that in a production cluster that it is recommended to have 5 nodes vs 3 as 4 nodes could cause split brain.