diff --git a/README.md b/README.md index c121d38..63dd1e0 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # opentera-webrtc-ros -Welcome to the opentera-webrtc-ros project. The goal of the project is to provide useful ROS nodes to stream audio/video/data through Google's WebRTC library wrapped in [opentera-webrtc](https://github.com/introlab/opentera-webrtc). Wrappers are written in C++ and Python and are compatible with ROS1 (Noetic) at the moment. We use the [signaling server](https://github.com/introlab/opentera-webrtc/tree/main/signaling-server) implementation provided by opentera-webrtc. +Welcome to the opentera-webrtc-ros project. The goal of the project is to provide useful ROS nodes to stream audio/video/data through Google's WebRTC library wrapped in [opentera-webrtc](https://github.com/introlab/opentera-webrtc). Wrappers are written in C++ and Python and are compatible with ROS2 (Humble) at the moment. Support for ROS1 (Noetic) is in the [`ros1`](https://github.com/introlab/opentera-webrtc-ros/tree/ros1) branch. We use the [signaling server](https://github.com/introlab/opentera-webrtc/tree/main/signaling-server) implementation provided by opentera-webrtc. Here are the key features: -* [ROS Messages](opentera_webrtc_ros_msgs) adding compatibility with ROS and [OpenTera protobuf protocol](https://github.com/introlab/opentera_messages) used in[opentera-teleop-service](https://github.com/introlab/opentera-teleop-service). +* [ROS Messages](opentera_webrtc_ros_msgs) adding compatibility with ROS and [OpenTera protobuf protocol](https://github.com/introlab/opentera_messages) used in [opentera-teleop-service](https://github.com/introlab/opentera-teleop-service). * [ROS Streaming nodes](opentera_webrtc_ros/README.md) capable of sending / receiving audio, video and data from WebRTC streams. @@ -14,7 +14,7 @@ Here are the key features: * Sound Source Localization / Tracking / Separation using [ODAS ROS](https://github.com/introlab/odas_ros). -* [Stand alone demonstrations](opentera_webrtc_demos/README.md) with simulated robot in gazebo. +* [Stand alone demonstrations](opentera_webrtc_demos/README.md) with simulated robot in Gazebo. * [Robot side front-end (Qt)](opentera_webrtc_robot_gui/README.md) to display call information and remote user interaction. @@ -42,48 +42,52 @@ The project is licensed with: ## Dependencies / Requirements -The procedure is written for Ubuntu 20.04 using ROS noetic. We assume ROS is already installed. If not, follow the [ROS Installation Instructions](http://wiki.ros.org/noetic/Installation/Ubuntu) first. The following packages must also be installed : +The procedure is written for Ubuntu 22.04 using ROS2 Humble. We assume ROS is already installed. If not, follow the [ROS2 Humble Installation Instructions](https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html) first. A more recent CMake than the default on Ubunuttu 22.04 is required. If the installed CMake version is 3.22, follow the [CMake Installation Instructions](https://apt.kitware.com/). The following packages must also be installed : ```bash +# utilities +sudo apt install unzip rsync ros-dev-tools + # opentera-webrtc-ros packages -sudo apt-get install nodejs ros-noetic-turtlebot3 ros-noetic-turtlebot3-gazebo ros-noetic-cv-camera ros-noetic-dwa-local-planner ros-noetic-rtabmap-ros +sudo apt install ros-humble-camera-info-manager ros-humble-rtabmap-ros ros-humble-rqt-tf-tree ros-humble-turtlebot3-gazebo ros-humble-turtlebot3-description ros-humble-turtlebot3-navigation2 ros-humble-joint-state-publisher-gui # protobuf -sudo apt-get install libprotobuf-dev protobuf-compiler python3-protobuf +sudo apt install libprotobuf-dev protobuf-compiler python3-protobuf # python dependencies -sudo apt-get install python3-pip portaudio19-dev +sudo apt install python3-pip portaudio19-dev # nodejs dependencies -sudo apt-get install nodejs npm +sudo apt install nodejs npm # audio_utils packages -sudo apt-get install cmake build-essential gfortran texinfo libasound2-dev libpulse-dev libgfortran-*-dev +sudo apt install cmake build-essential gfortran texinfo libasound2-dev libpulse-dev 'libgfortran-*-dev' # odas_ros packages -sudo apt-get install libfftw3-dev libconfig-dev +sudo apt install libfftw3-dev libconfig-dev + +# gstreamer for hardware acceleration +sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools # qt submodules -sudo apt-get install libqt5charts5-dev +sudo apt install libqt5charts5-dev ``` ## Installation -### 1 - Create a catkin workspace (if not already done) +### 1 - Create a colcon workspace (if not already done) ```bash -# Make sure ROS is installed first. -source /opt/ros/noetic/setup.bash -# Create the workspace and initial build files +# Create the workspace mkdir -p ~/teleop_ws/src -cd ~/teleop_ws/ -catkin_make ``` ### 2 - Get all the required ROS packages ```bash cd ~/teleop_ws/src +# cv_camera +git clone https://github.com/Kapernikov/cv_camera.git # audio_utils git clone https://github.com/introlab/audio_utils.git --recurse-submodules # odas_ros @@ -107,8 +111,7 @@ python3 -m pip install -r requirements.txt ```bash cd ~/teleop_ws -source devel/setup.bash -catkin_make +colcon build --symlink-install --cmake-args -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCMAKE_BUILD_TYPE=Debug --no-warn-unused-cli ``` ## Running the demos diff --git a/face_cropping/README.md b/face_cropping/README.md index b12ff48..69f236b 100644 --- a/face_cropping/README.md +++ b/face_cropping/README.md @@ -47,9 +47,9 @@ It can be enabled and disabled from the robot GUI by pressing the face cropping ### Subscribed Topics -- `enable_face_cropping` [std_msgs/Bool](http://docs.ros2.org/foxy/api/std_msgs/msg/Bool.html): The topic to enable or disable the node. -- `input_image` [sensor_msgs/Image](http://docs.ros2.org/foxy/api/sensor_msgs/msg/Image.html): The input image topic. +- `enable_face_cropping` [std_msgs/Bool](https://docs.ros.org/en/humble/p/std_msgs/interfaces/msg/Bool.html): The topic to enable or disable the node. +- `input_image` [sensor_msgs/Image](https://docs.ros.org/en/humble/p/sensor_msgs/interfaces/msg/Image.html): The input image topic. ### Published Topics -- `output_image` [sensor_msgs/Image](http://docs.ros2.org/foxy/api/sensor_msgs/msg/Image.html): The output image topic. +- `output_image` [sensor_msgs/Image](https://docs.ros.org/en/humble/p/sensor_msgs/interfaces/msg/Image.html): The output image topic. diff --git a/map_image_generator/package.xml b/map_image_generator/package.xml index 3bca25b..5b19ac4 100644 --- a/map_image_generator/package.xml +++ b/map_image_generator/package.xml @@ -18,7 +18,7 @@ visualization_msgs rtabmap_msgs opentera_webrtc_ros_msgs - odas_ros + odas_ros_msgs cv_bridge framework diff --git a/opentera_webrtc_demos/README.md b/opentera_webrtc_demos/README.md index e67e619..a325c36 100644 --- a/opentera_webrtc_demos/README.md +++ b/opentera_webrtc_demos/README.md @@ -4,21 +4,17 @@ To launch the stand alone demo try : ```bash # Load ROS environment -$ source ~/teleop_ws/install/setup.bash -# Needed for simulation -$ export TURTLEBOT3_MODEL=waffle +source ~/teleop_ws/install/setup.bash # Run the demo -$ ros2 launch opentera_webrtc_demos demo.launch is_stand_alone:=true +ros2 launch opentera_webrtc_demos demo.launch.xml is_stand_alone:=true ``` To launch the stand alone demo with ODAS functionnality, you will need the [ReSpeaker 4-mic array](https://respeaker.io/usb_4_mic_array/) plugged-in your computer: ```bash # Load ROS environment -$ source ~/teleop_ws/install/setup.bash -# Needed for simulation -$ export TURTLEBOT3_MODEL=waffle +source ~/teleop_ws/install/setup.bash # Run the demo -$ ros2 launch opentera_webrtc_demos demo_odas.launch is_stand_alone:=true +ros2 launch opentera_webrtc_demos demo_odas.launch.xml is_stand_alone:=true ``` Once launched go to the following URL : diff --git a/opentera_webrtc_robot_gui/src/MainWindow.cpp b/opentera_webrtc_robot_gui/src/MainWindow.cpp index 80c3505..7b7b76b 100644 --- a/opentera_webrtc_robot_gui/src/MainWindow.cpp +++ b/opentera_webrtc_robot_gui/src/MainWindow.cpp @@ -104,7 +104,7 @@ void MainWindow::localImageCallback(const sensor_msgs::msg::Image::ConstSharedPt if (msg->encoding == "rgb8") { // Step #1 Transform ROS Image to QtImage - QImage image(&msg->data[0], msg->width, msg->height, QImage::Format_RGB888); + QImage image(&msg->data[0], msg->width, msg->height, msg->step, QImage::Format_RGB888); // Step #2 emit new signal with image emit newLocalImage(image.copy()); @@ -112,7 +112,7 @@ void MainWindow::localImageCallback(const sensor_msgs::msg::Image::ConstSharedPt else if (msg->encoding == "bgr8") { // Step #1 Transform ROS Image to QtImage - QImage image(&msg->data[0], msg->width, msg->height, QImage::Format_RGB888); + QImage image(&msg->data[0], msg->width, msg->height, msg->step, QImage::Format_RGB888); // Step #2 emit new signal with image // Invert R & B here @@ -239,7 +239,7 @@ void MainWindow::closeCameraWindow() void MainWindow::peerImageCallback(const opentera_webrtc_ros_msgs::msg::PeerImage::ConstSharedPtr& msg) { // Step #1 Transform ROS Image to QtImage - QImage image(&msg->frame.data[0], msg->frame.width, msg->frame.height, QImage::Format_RGB888); + QImage image(&msg->frame.data[0], msg->frame.width, msg->frame.height, msg->frame.step, QImage::Format_RGB888); // Step #2 Emit signal (will be handled in Qt main thread) // Image will be automatically deleted when required diff --git a/opentera_webrtc_ros/CMakeLists.txt b/opentera_webrtc_ros/CMakeLists.txt index 946f0d3..b14cc3a 100644 --- a/opentera_webrtc_ros/CMakeLists.txt +++ b/opentera_webrtc_ros/CMakeLists.txt @@ -37,7 +37,7 @@ find_package(std_msgs REQUIRED) find_package(std_srvs REQUIRED) find_package(sensor_msgs REQUIRED) find_package(geometry_msgs REQUIRED) -find_package(audio_utils REQUIRED) +find_package(audio_utils_msgs REQUIRED) find_package(opentera_webrtc_ros_msgs REQUIRED) find_package(map_image_generator REQUIRED) @@ -145,7 +145,7 @@ ament_target_dependencies(${PROJECT_NAME}_topic_streamer std_srvs sensor_msgs geometry_msgs - audio_utils + audio_utils_msgs ) target_link_libraries(${PROJECT_NAME}_topic_streamer ${opentera_webrtc_native_client_libs} @@ -167,7 +167,7 @@ ament_target_dependencies(${PROJECT_NAME}_data_channel_bridge std_srvs sensor_msgs geometry_msgs - audio_utils + audio_utils_msgs ) target_link_libraries(${PROJECT_NAME}_data_channel_bridge ${opentera_webrtc_native_client_libs} @@ -188,7 +188,7 @@ ament_target_dependencies(${PROJECT_NAME}_json_data_handler std_srvs sensor_msgs geometry_msgs - audio_utils + audio_utils_msgs ) set_target_properties(${PROJECT_NAME}_json_data_handler PROPERTIES OUTPUT_NAME json_data_handler PREFIX "") install(TARGETS ${PROJECT_NAME}_json_data_handler DESTINATION lib/${PROJECT_NAME}) diff --git a/opentera_webrtc_ros/README.md b/opentera_webrtc_ros/README.md index 8201dfb..7b0b251 100644 --- a/opentera_webrtc_ros/README.md +++ b/opentera_webrtc_ros/README.md @@ -19,14 +19,14 @@ It also forwards images and audio received on the WebRTC stream to ROS. #### Subscribes -- ros_image : `sensor_msgs::Image` -- audio_in : `audio_utils::AudioFrame` +- ros_image : `sensor_msgs/Image` +- audio_in : `audio_utils_msgs/AudioFrame` #### Advertises -- webrtc_image : `opentera_webrtc_ros::PeerImage` -- webrtc_audio : `opentera_webrtc_ros::PeerAudio` -- audio_mixed : `audio_utils::AudioFrame` +- webrtc_image : `opentera_webrtc_ros_msgs/PeerImage` +- webrtc_audio : `opentera_webrtc_ros_msgs/PeerAudio` +- audio_mixed : `audio_utils_msgs/AudioFrame` #### Default Parameters @@ -40,7 +40,7 @@ It also forwards images and audio received on the WebRTC stream to ROS. - + @@ -48,7 +48,7 @@ It also forwards images and audio received on the WebRTC stream to ROS. ``` -For usage exemple look at [ros_stream_client.launch](launch/ros_stream_client.launch). +For usage exemple look at [ros_stream_client.launch.xml](launch/ros_stream_client.launch.xml). ## RosDataChannelBridge @@ -59,17 +59,17 @@ data channel. It also forwards messages received on the WebRTC data channel to R #### Subscribes (RosDataChannelBridge) -- ros_data : `std_msgs::String` +- ros_data : `std_msgs/String` #### Advertises (RosDataChannelBridge) -- webrtc_data : `opentera_webrtc_ros_msgs::PeerData` +- webrtc_data : `opentera_webrtc_ros_msgs/PeerData` #### Default Parameters (RosDataChannelBridge) ```xml - + @@ -77,7 +77,7 @@ data channel. It also forwards messages received on the WebRTC data channel to R ``` -For usage exemple look at [ros_data_channel_client.launch](launch/ros_data_channel_client.launch). +For usage exemple look at [ros_data_channel_client.launch.xml](launch/ros_data_channel_client.launch.xml). ## RosJsonDataHandler @@ -87,21 +87,21 @@ Implement a ROS node that dispatch received JSON messages and forward them on th #### Subscribes (RosJsonDataHandler) -- webrtc_data : `opentera_webrtc_ros_msgs::PeerData` +- webrtc_data : `opentera_webrtc_ros_msgs/PeerData` #### Advertises (RosJsonDataHandler) -- cmd_vel : `geometry_msgs::Twist` +- cmd_vel : `geometry_msgs/Twist` #### JSON Format (RosJsonDataHandler) -For usage exemple look at [ros_json_data_handler.launch](launch/ros_json_data_handler.launch). +For usage exemple look at [ros_json_data_handler.launch.xml](launch/ros_json_data_handler.launch.xml). ## libnavigation ### Description (libnavigation) -Shared library for `goal_manager` and `label_manager` to send navigation commands to `move_base` as well as navigation waypoints to `map_image_generator`. +Shared library for `goal_manager` and `labels_manager` to send navigation commands to the `nav2` stack as well as navigation waypoints to `map_image_generator`. The library is embedded in both nodes, a single instance is not shared. #### Subscribed topics (libnavigation) @@ -138,7 +138,7 @@ Manages labels. The stored labels are dependent on the `map` transform and the database needs to be cleaned if the map changes. A label represents a name grouped with an associated pose and a description. This node relies on a service provided by the `map_image_generator` package to convert the labels from image coordinates to map coordinates. -This node can also send a label as a goal to `move_base` by its name. +This node can also send a label as a goal to `nav2` by its name. #### Subscribed topics (labels_manager) diff --git a/opentera_webrtc_ros/package.xml b/opentera_webrtc_ros/package.xml index c70689a..dd2c50a 100644 --- a/opentera_webrtc_ros/package.xml +++ b/opentera_webrtc_ros/package.xml @@ -18,7 +18,7 @@ std_msgs std_srvs sensor_msgs - audio_utils + audio_utils_msgs opentera_webrtc_ros_msgs map_image_generator diff --git a/opentera_webrtc_ros_msgs/CMakeLists.txt b/opentera_webrtc_ros_msgs/CMakeLists.txt index 759bc46..8ede306 100644 --- a/opentera_webrtc_ros_msgs/CMakeLists.txt +++ b/opentera_webrtc_ros_msgs/CMakeLists.txt @@ -5,7 +5,7 @@ find_package(ament_cmake REQUIRED) find_package(std_msgs REQUIRED) find_package(geometry_msgs REQUIRED) find_package(sensor_msgs REQUIRED) -find_package(audio_utils REQUIRED) +find_package(audio_utils_msgs REQUIRED) find_package(rosidl_default_generators REQUIRED) rosidl_generate_interfaces(${PROJECT_NAME} @@ -38,7 +38,7 @@ rosidl_generate_interfaces(${PROJECT_NAME} "srv/ImageGoalToMapGoal.srv" "srv/SetString.srv" - DEPENDENCIES std_msgs geometry_msgs sensor_msgs audio_utils + DEPENDENCIES std_msgs geometry_msgs sensor_msgs audio_utils_msgs ) ament_package() diff --git a/opentera_webrtc_ros_msgs/package.xml b/opentera_webrtc_ros_msgs/package.xml index 7efbcdf..9beddf3 100644 --- a/opentera_webrtc_ros_msgs/package.xml +++ b/opentera_webrtc_ros_msgs/package.xml @@ -15,7 +15,7 @@ std_msgs geometry_msgs sensor_msgs - audio_utils + audio_utils_msgs rosidl_interface_packages