How to Reduce Streaming Lag Caused by Slow Processing?

[jakobhec]

In this example, the processing function takes 200ms per frame, but the input stream runs at 30 FPS. This means new frames arrive faster than they can be processed, causing increasing delay.

Is there a standard approach for handling this? Are there any examples?

A simple solution could be to always process only the latest input. This would make the output video choppy but keep it in sync.

import time

import numpy as np
from fastrtc import Stream


def process_image(image):
    time.sleep(0.2)  # Simulating 200ms processing time per frame; input arrives faster (30 FPS).
    return np.flip(image, axis=0)


stream = Stream(
    handler=process_image,
    modality='video',
    mode='send-receive',
)

stream.ui.launch()

[freddyaboulton]

Hi @jakobhec ! I'm not sure what we can change. The delay is not caused because the input frames arrive faster than the output frames, we have a queuing mechanism that applies backpressure so the input frames are processed as fast as output frames are produced. So if outputting a frame takes .2 seconds, the output stream will naturally have 5 fps.

Let me know if I'm thinking about this incorrectly.

[freddyaboulton]

Some other thoughts:

• Can you speed up the processing? In the object_detection demo in the demo directory, we use an onnx model to keep the inference very fast.
• Instead of making it a "live" demo, you can have the user upload a video or record one with their webcam and then process every 20 frames instead of all the frames. Or process them in batches and then yield a batch successively.

[jakobhec]

Hi @freddyaboulton! Could you please explain a bit more what you mean by a “queuing mechanism that applies backpressure”?

Here’s how I understand it:
If the input stream runs at 30 FPS (one frame every 33ms), but each frame takes 200ms to process, the delay keeps growing over time. After 1 second, the webcam has sent 30 frames, but processing those frames takes 6 seconds, leading to a 5-second delay. After 10 seconds, 300 frames have arrived, requiring 60 seconds to process, resulting in a 50-second delay. So it grows over time. And yes, the output stream runs at 5 FPS.

I would like to avoid optimizing inference performance during prototyping, and I want to use it on a live webcam stream rather than a video.

I would love to have an easy option to pass only the latest frame to the handler and drop all intermediate frames. This way, the output stream would still run at 5 FPS but stay in sync with the camera without an increasing delay.