Debug DepthAI ros driver crashing camera

[nicholaschan23]

The node doesn't crash and is still up. However the camera doesn't seem to be recording anymore and continuously outputs this message:

[component_container-1] [ERROR] [1733870724.740647293] [oak]: No data on logger queue!
[component_container-1] [ERROR] [1733870724.740984478] [oak]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'

[nicholaschan23]

Sometimes on start launching the driver, it errors and the node doesn't even start.

[component_container-1] terminate called after throwing an instance of 'dai::XLinkWriteError' [component_container-1] what(): Couldn't write data to stream: '__stream_asset_storage' (X_LINK_ERROR)

But usually when you wait a little and rerun the same launch script, it works. and says 'Camera ready!` at the end.

[nicholaschan23]

Hi, this might be due to high CPU usage on the camera, you can monitor it either via diagnostics topic or by enabling debug information - export DEPTHAI_DEBUG=1, you can also set a parameter camera_i_restart_on_diagnostics_error: true to have the camera restart after encountering an error

luxonis/depthai-ros#575

[nicholaschan23]

Currently I just either:

• Unplug and replug and launch the driver again
• If that doesn't work, reboot the robot

[nicholaschan23]

Couldn't read data from stream: 'rgb' (X_LINK_ERROR)
luxonis/depthai-python#318

[nicholaschan23]

Randomly crashes when running.

[component_container-1] [ERROR] [1736877275.054611606] [oak]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'
[component_container-1] [ERROR] [1736877276.054121632] [oak]: No data on logger queue!

[nicholaschan23]

Fails to boot.

[INFO] [launch]: All log files can be found below /home/nickchan/.ros/log/2025-01-14-10-45-34-071570-nrgadmin-Nuvo-9000-Series-7268
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [7283]
[component_container-1] [INFO] [1736873134.475605287] [oak_container]: Load Library: /opt/ros/humble/lib/librobot_state_publisher_node.so
[component_container-1] [INFO] [1736873134.480345434] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<robot_state_publisher::RobotStatePublisher>
[component_container-1] [INFO] [1736873134.480391689] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<robot_state_publisher::RobotStatePublisher>
[component_container-1] [INFO] [1736873134.520505357] [oak_state_publisher]: got segment oak
[component_container-1] [INFO] [1736873134.520532499] [oak_state_publisher]: got segment oak-d-base-frame
[component_container-1] [INFO] [1736873134.520538896] [oak_state_publisher]: got segment oak_imu_frame
[component_container-1] [INFO] [1736873134.520543230] [oak_state_publisher]: got segment oak_left_camera_frame
[component_container-1] [INFO] [1736873134.520547449] [oak_state_publisher]: got segment oak_left_camera_optical_frame
[component_container-1] [INFO] [1736873134.520551298] [oak_state_publisher]: got segment oak_model_origin
[component_container-1] [INFO] [1736873134.520554928] [oak_state_publisher]: got segment oak_rgb_camera_frame
[component_container-1] [INFO] [1736873134.520558642] [oak_state_publisher]: got segment oak_rgb_camera_optical_frame
[component_container-1] [INFO] [1736873134.520562545] [oak_state_publisher]: got segment oak_right_camera_frame
[component_container-1] [INFO] [1736873134.520566285] [oak_state_publisher]: got segment oak_right_camera_optical_frame
[component_container-1] [INFO] [1736873134.521630445] [oak_container]: Load Library: /home/nickchan/utexas_panther_ws/install/depthai_ros_driver/lib/libdepthai_ros_driver.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/oak_state_publisher' in container '/oak_container'
[component_container-1] [INFO] [1736873134.567378714] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1736873134.567415441] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1736873134.569305171] [oak_container]: Load Library: /opt/ros/humble/lib/librectify.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/oak' in container '/oak_container'
[component_container-1] [INFO] [1736873134.570756566] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<image_proc::RectifyNode>
[component_container-1] [INFO] [1736873134.570770455] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<image_proc::RectifyNode>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/rectify_color_node' in container '/oak_container'
[component_container-1] [INFO] [1736873135.572337716] [oak]: Starting camera.
[component_container-1] [INFO] [1736873135.578414378] [oak]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [18443010D109A90F00] [1.14] [1736873149.477] [host] [warning] Device crashed, but no crash dump could be extracted.
[component_container-1] [ERROR] [1736873150.981071952] [oak]: Device already closed or disconnected: Input/output error
[component_container-1] [INFO] [1736873150.981176080] [oak]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [18443010D109A90F00] [1.14] [1736873165.535] [host] [warning] Device crashed, but no crash dump could be extracted.
[component_container-1] [ERROR] [1736873167.039392975] [oak]: Device already closed or disconnected: Input/output error
[component_container-1] [INFO] [1736873167.039483724] [oak]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [INFO] [1736873170.429618152] [oak]: Camera with MXID: 18443010D109A90F00 and Name: 1.14 connected!
[component_container-1] [INFO] [1736873170.430392345] [oak]: USB SPEED: HIGH
[component_container-1] [INFO] [1736873170.477045606] [oak]: Device type: OAK-D-W
[component_container-1] [INFO] [1736873170.479470924] [oak]: Pipeline type: RGBD
[component_container-1] [INFO] [1736873170.928454035] [oak]: NN Family: mobilenet
[component_container-1] [INFO] [1736873170.967594974] [oak]: NN input size: 300 x 300. Resizing input image in case of different dimensions.
[component_container-1] [INFO] [1736873171.525300848] [oak]: Finished setting up pipeline.
[component_container-1] terminate called after throwing an instance of 'std::system_error'
[component_container-1]   what():  Device already closed or disconnected: Input/output error
[ERROR] [component_container-1]: process has died [pid 7283, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container --ros-args --log-level info --ros-args -r __node:=oak_container -r __ns:=/'].