Cell Error timeout on request ['heartbeat'] and Client send heartbeat

[plaipmc]

I try the PyTorch nvflare example after run set_env.sh and hw_pre_start.sh on WSL (Ubuntu 22.04.2 TLS) with nvflare version 2.3.5 and I got

2023-10-11 13:53:34,989 - Cell - ERROR - [ME=site-2 O=? D=server F=? T=? CH=task TP=heart_beat] timeout on Request 658ac7f4-3964-4ace-b626-58490b587c5f for ['heart_beat'] after 30.0 secs
2023-10-11 13:53:55,065 - Cell - ERROR - [ME=site-1 O=? D=server F=? T=? CH=task TP=heart_beat] timeout on Request 83199230-6e23-40f3-9418-71b6f7cc2027 for ['heart_beat'] after 30.0 secs
2023-10-11 13:55:42,563 - Communicator - INFO - Client: site-2 has sent 30 heartbeats.
2023-10-11 13:56:14,379 - Communicator - INFO - Client: site-1 has sent 30 heartbeats.

And after 1 min (approximately), It continues the job and got the heartbeat again like this.

2023-10-11 14:07:13,966 - Communicator - INFO - Client: site-1 has sent 40 heartbeats.
2023-10-11 14:08:24,132 - Communicator - INFO - Client: site-2 has sent 50 heartbeats.
2023-10-11 14:08:54,188 - Communicator - INFO - Client: site-1 has sent 50 heartbeats.
2023-10-11 14:10:04,306 - Communicator - INFO - Client: site-2 has sent 60 heartbeats.
2023-10-11 14:10:34,373 - Communicator - INFO - Client: site-1 has sent 60 heartbeats.

So I want to know what going on and why it happen? also what can I fix it to run more smooth

Thank you so much.

[YuanTingHsieh]

Hi @plaipmc thanks for your interest!

Please provide your NVFlare branch. (You can use "git branch")
Please also provide us the steps you have done.

Was the job working properly and finished?
If that was the case, then this message "Cell - ERROR - [ME=site-1 O=? D=server F=? T=? CH=task TP=heart_beat] timeout on Request 83199230-6e23-40f3-9418-71b6f7cc2027 for ['heart_beat'] after 30.0 secs" can be safely ignored.
It only means that one of the heart beat request times out

[YuanTingHsieh]

@plaipmc Hi I looked at your logs again and notice:

2023-10-24 17:47:01,272 - SimulatorClientRunner - ERROR - run_client_thread error: EOFError:

This means that your "site-1" simulating process/threads disappeared.
I suspect the OS kill it.
Can you please check your CPU memory usage?

In this example, we specify min_clients=2.
After server receives result from site-2,
server is waiting for result from site-1
BUT site-1 is killed somehow so it never returns
Thus the server waits a long long time in your case.

[plaipmc]

I think you're right. When the workflow stuck, CPU reach 100% for a few time. Maybe CPU kill the client.

Can you suggest me more how can I fix it?? Thank you so much.

[YuanTingHsieh]

@plaipmc we can't really do much if the resource is not enough.

I will suggest the following:

1. If just want to get things run and see some proof-of-concept results -> reduce the model network size and reduce the dataset size, this should reduce memory usage
2. If want to run real world examples later -> NEED to find a larger machine with better CPU or adding more CPU memory, even add a GPU for faster execution

[plaipmc]

Thank you so much and I think we have the answer for this issue.

1. Heartbeat issue: just not run . hw_pre_start.sh before run simulator
2. Workflow stop: resource issues can be fix by reduce model size or require better CPU / add memory and GPU.

Anyway, Can we add the GPU for compute and how? Thank you.

[YuanTingHsieh]

@plaipmc you can find plenty of resources online on how to setup GPUs with CUDA and nvidia drivers.

Once you set that up, the PyTorch/Tensorflow framework is going to pick it up automatically.