Measuring Round Trip Latency (messages are sometimes delayed for 4 seconds)

[mechaniputer]

Hi everyone,

I am attempting to accurately measure the round trip latency of FastDDS. I have a pair of machines connected with 10 gigabit ethernet.

The first machine runs two processes. A driver and a listener. The driver sends samples that contain timestamps. These are sent on a topic to which another participant/process on the second machine is subscribed. This second process is called the "server". It immediately relays every sample it receives back on a second topic, to which a third participant/process on the first machine is subscribed. This is the "listener" process. It measures the difference between the received timestamp and the current time, and calculates the average latency over the course of a test run.

I was getting unexpectedly high average latency, sometimes around 30 ms. I started printing out the latency of every sample and I am getting numbers around ~200000 nanoseconds for most of them. However in every test, some number of samples show a latency in excess of 4 seconds! This skews the average dramatically.

Here is my code: https://github.com/mechaniputer/LatencyBench-FastDDS

I have two questions.

1. Why is this happening
2. How do I make it stop happening and get useful stats on the latency in this test?

Thank you.

[mechaniputer]

Some additional info. The QoS settings can be found in the code repo I linked, but in summary:

driver_qos.reliability().kind = RELIABLE_RELIABILITY_QOS;
driver_qos.reliability().max_blocking_time = eprosima::fastrtps::Duration_t(2,0); // 2 sec
driver_qos.reliable_writer_qos().times.heartbeatPeriod.seconds = 0;
driver_qos.reliable_writer_qos().times.heartbeatPeriod.nanosec = 500000; // 500,000ns => 2000 Hz
driver_qos.history().kind = KEEP_ALL_HISTORY_QOS;
driver_qos.durability().kind = TRANSIENT_LOCAL_DURABILITY_QOS;

And here are some additional observations which may help identify the issue:

1. When the server is run on the 2nd machine, some samples have a very extended latency of around 4 seconds, but this stops happening if all processes are run locally on the same machine.

2. Regardless of whether all processes are on the same machine or not, I also observe latency spikes to 0.01 seconds every so often, and these also affect the average.

3. Currently, when I run all 3 processes on the same machine, the latencies are still mostly around ~200000 ns, similar to when I use the network. This worries me because I would expect it to be much faster.

Edit: It looks like the 4 second spikes become far less frequent if I remember to source Fast-DDS/install/setup.bash (oops). Still seeing occasional spikes to both 4s and 0.01s, I'm also curious what sourcing that file does that changes the performance in this way, and still wondering why the non-networked latency appears to be the same as networked in my current configuration.