Ozone OM HA not working with Hadoop 3.3.6 Client

[ashtippin]

I recently setup a Ozone HA cluster running in a Kubernetes environment. When trying to connect to the OM using a hadoop client over the ofs client interface , I am getting the following error

RetryProxy: OM om1: RemoteException: OM:om-0 is not the leader. Suggested leader is OM:om-2[om-2.om.ozone-test.svc.cluster.local/10.32.11.219].
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:292)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:274)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:267)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.internalProcessRequest(OzoneManagerProtocolServerSideTranslatorPB.java:211)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:171)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:162)
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:484)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)

I do not encounter this error when running OM with a single node, it only appears when Ratis is enabled for the OM. If anyone has any suggestions on what can be done to resolve this issue, it would be greatly appreciated.

[jojochuang]

did your query/command fail? This message may be benign if it appears only once per query.

[jojochuang]

if this message appears many times, and that your command appears to hang, and the host names in the message "OM:om-0 is not the leader. Suggested leader is OM:om-2" keeps changing, then you may have a problem in the configuration. Make sure client and Ozone Manager have the same OM URI.

[ashtippin]

Yes, my query fails. I also have copied the config directly from OM into my client, so the OM uri is the same.

[jojochuang]

The stack trace in the OP is OM side.
Would you mind posting the error message & stacktraces at the client side? I suspect the failure isn't related to this message.

[ashtippin]

So the stack trace I posted is actually showing up on the client side, when trying to run a hdfs command. This is the command I run, and the result.

[root@accumulo-master-0 bin]# ./hdfs dfs -ls /
2024-04-09 18:52:02,540 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:om-0 is not the leader. Suggested leader is OM:om-2[om-2.om.ozone-test.svc.cluster.local/10.42.6.230].
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:292)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:274)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:267)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.internalProcessRequest(OzoneManagerProtocolServerSideTranslatorPB.java:211)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:171)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:162)
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:484)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:595)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:573)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1227)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1094)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1017)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3048)
, while invoking $Proxy13.submitRequest over nodeId=null,nodeAddress=om:9862 after 1 failover attempts. Trying to failover after sleeping for 4000ms. Current retry count: 1.
2024-04-09 18:52:06,551 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:om-0 is not the leader. Suggested leader is OM:om-2[om-2.om.ozone-test.svc.cluster.local/10.42.6.230].

The client is making a request to the remote filesystem, as declared in the core-site.xml as

 <property>
            <name>fs.defaultFS</name>
            <value>ofs://om/</value>
</property>

where om is the name of a headless service that is pointed to the 3 Ratis enabled OM pods.

[errose28]

What is the result of ozone admin om roles? If this indicates a leader node, then the issue is the client identifying the leader. If this says that all the nodes are followers, then the issue is the OMs are unable to finish a leader election and further troubleshooting would involve looking into the OM logs for ratis log messages to determine why leader elections are failing.

[ashtippin]

the result of ozone admin om roles shows that there is indeed a leader node. The client is unable to identify the leader. I am using a hadoop 3.3.6 client and just trying to run hdfs commands from the client pod. From what the documentation says this should be totally possible as long as the compatability jars are added to the classpath of client pod. However, I havent had any luck getting it to work with a HA OM configuration.

[jojochuang]

what's your version of Ozone?
the hadoop 3.3.6 client jar shouldn't be a problem. we use hadoop 3.3.6 in Ozone 1.4 and master branch (1.5.0-SNAPSHOT)

[ashtippin]

We are using Ozone 1.4

[ChenSammi]

Can you try with "./hdfs --loglevel DEBUG dfs -ls /" and share the console output if possible?