[20291] Gather-send implementation #4537

cferreiragonz · 2024-03-08T12:25:33Z

Description

This PR refactors the creation and sending of RTPS Messages by utilizing a list of NetworkBuffers (pointers + sizes) as the data transmission object, thereby eliminating an additional data message copy in the RTPSMessageGroup.
This refactor impacts both the Transports and the RTPS Layer, involving updates to the sender resources lambda functions, transport interfaces, statistics module, and the logic implemented in RTPS message creation.

Two extra ResourceLimitedVectors have been added to RTPSMessageGroup_t to act as NetworkBuffers and to contain the metadata of the payloads used in the creation of the RTPS message. In this way mallocs can be reduced as much as possible during runtime. TCP transport has mallocs during the creation of the TCP message and during the asio write operation.

SendBuffersAllocationAttributes has been extended with a new attribute defining the allocation configuration of the NetworkBuffers.

This PR is on top of: [21121] IPayloadPool refactor #4892 and needs to be merged after it, in order to use the latest changes of payload_owner_

Opened as draft:

Refactor at Transport level
Refactor at Sender Resource level
Refactor at Participant level
Statistics
Redesign on how secure messages are managed
Doxygen
Tests. They have been updated with new API. Gather-send is implicitly tested in every other send.
Documentation

Contributor Checklist

Commit messages follow the project guidelines.
The code follows the style guidelines of this project.
[N/A] Tests that thoroughly check the new feature have been added/Regression tests checking the bug and its fix have been added; the added tests pass locally
Any new/modified methods have been properly documented using Doxygen.
❌ Changes are ABI compatible.
❌ Changes are API compatible.
New feature has been added to the versions.md file (if applicable).
New feature has been documented/Current behavior is correctly described in the documentation.
Related documentation PR: [20291] Gather-send implementation Docs Fast-DDS-docs#724
[N/A] Applicable backports have been included in the description.

Reviewer Checklist

The PR has a milestone assigned.
The title and description correctly express the PR's purpose.
Check contributor checklist is correct.
Check CI results: changes do not issue any warning.
Check CI results: failing tests are unrelated with the changes.

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

MiguelCompany

A nit that does not require to pass CI again

test/blackbox/common/DDSBlackboxTestsBasic.cpp

MiguelCompany · 2024-06-13T06:02:52Z

@richiprosima Please test_3 discovery-server

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

* Refs #20337: Buffer list in UDP sending function Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20337: Buffer list in SHM sending function & Copy to shared buffer function Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20337: Buffer list in TCP sending function Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Add new Buffer structure Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Add new lambda to send buffers into SenderResources Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on UDP transport Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on UDP test_transport Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on TCP Transport Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on TCPChannelResource Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on TCP tests Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on SHM transport & enable copying multiple buffers Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Refactor on SHM tests Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20342: Fix mock tests after rebase Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on addSubmessageData/DataFrag Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Add constructor overloads to NetworkBuffers Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Add new attributes and methods to RTPSMessageGroup.h Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on add_data() Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on add_data_frag() Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on insert_submessage() Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on send() Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Minor changes in RTPSMessageGroup Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on RTPSMessageSenderInterface Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on ChaningTransport and ABI compatible send_lambda_ Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Refactor on Statistics module Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Add security support Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Minor fixes Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Doxygen Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Delete Sender's Resource deprecated API Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Uncrustify Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Fix Windows build Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Account for change of namespaces Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Add NetworkBuffer.cpp Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Revision minor changes Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20352: Revision minor changes 2 Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Avoid stats_msg dynamic malloc Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Use vector instead of list Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Refactor RTPSMessageGroup to avoid Mallocs Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Use limited vector to avoid repeated mallocs Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Fix rebase Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Improve doxygen Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Revision Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Add ResourceLV config into QoS Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: XML - New QoS added Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Get Payload in RTPSMessageGroup Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Uncrustify Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Revision - Use RLContainerConfig and minor changes Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Revision - Get payload after correct RTPSMsg creation Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Revision - Default values Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Update versions.md & CMakeLists Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Revision - Headers & versions.md Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Adjust payload_pool test Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> * Refs #20291: Test comment Signed-off-by: cferreiragonz <carlosferreira@eprosima.com> --------- Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

cferreiragonz added this to the v3.0.0 milestone Mar 11, 2024

cferreiragonz force-pushed the poc/gather-send-implementation branch 2 times, most recently from aa6e6cc to 5de9ef4 Compare March 11, 2024 14:32

cferreiragonz marked this pull request as ready for review March 11, 2024 14:34

cferreiragonz force-pushed the poc/gather-send-implementation branch from 02e2919 to af3951c Compare March 12, 2024 07:12

cferreiragonz mentioned this pull request Mar 12, 2024

[20291] Gather-send implementation Docs eProsima/Fast-DDS-docs#724

Merged

7 tasks

cferreiragonz force-pushed the poc/gather-send-implementation branch 2 times, most recently from 376e650 to 261c9a0 Compare March 12, 2024 09:17

cferreiragonz changed the base branch from master to 3.0.x-devel March 12, 2024 10:17

cferreiragonz force-pushed the poc/gather-send-implementation branch from 261c9a0 to 3067287 Compare March 12, 2024 10:18

cferreiragonz added the needs-review PR that is ready to be reviewed label Mar 12, 2024

cferreiragonz force-pushed the poc/gather-send-implementation branch 3 times, most recently from 86c7d43 to c3741e5 Compare March 14, 2024 06:59

JLBuenoLopez force-pushed the 3.0.x-devel branch from 2de726b to 4a61143 Compare March 14, 2024 07:55

cferreiragonz force-pushed the poc/gather-send-implementation branch from c3741e5 to 6752776 Compare March 14, 2024 08:49

EduPonz force-pushed the 3.0.x-devel branch from 4a61143 to 39b2115 Compare March 20, 2024 09:15

cferreiragonz force-pushed the poc/gather-send-implementation branch from 6752776 to 9344ae3 Compare March 20, 2024 09:34

EduPonz force-pushed the 3.0.x-devel branch from 1f2e8b7 to 30d4fcd Compare March 21, 2024 07:18

cferreiragonz force-pushed the poc/gather-send-implementation branch from 9344ae3 to 3ff583d Compare March 22, 2024 07:39

EduPonz force-pushed the 3.0.x-devel branch 2 times, most recently from 8193fef to a364ed8 Compare March 27, 2024 14:54

github-actions bot force-pushed the 3.0.x-devel branch from 18d8c6e to f3b3af6 Compare April 3, 2024 00:04

richiware force-pushed the 3.0.x-devel branch from fb8e560 to 943b988 Compare April 11, 2024 13:01

JesusPoderoso force-pushed the 3.0.x-devel branch from 943b988 to 2c47b5e Compare April 12, 2024 05:50

github-actions bot force-pushed the 3.0.x-devel branch from 2c47b5e to c978d52 Compare April 16, 2024 08:56

EduPonz force-pushed the 3.0.x-devel branch 3 times, most recently from 5148f5d to bd83bb2 Compare April 23, 2024 05:52

cferreiragonz added the needs rebase label Apr 23, 2024

cferreiragonz added 20 commits June 13, 2024 07:58

Refs #20352: Add NetworkBuffer.cpp

d6a4146

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20352: Revision minor changes

6e83cc3

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20352: Revision minor changes 2

17cc6ec

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Avoid stats_msg dynamic malloc

a748ae4

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Use vector instead of list

809ae5d

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Refactor RTPSMessageGroup to avoid Mallocs

557c2ca

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Use limited vector to avoid repeated mallocs

153156b

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Fix rebase

ffaced5

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Improve doxygen

42a51dd

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Revision

4d9addc

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Add ResourceLV config into QoS

4134e23

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: XML - New QoS added

020b04d

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Get Payload in RTPSMessageGroup

c7ce07d

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Uncrustify

f84a62f

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Revision - Use RLContainerConfig and minor changes

32158d3

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Revision - Get payload after correct RTPSMsg creation

a5d7bee

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Revision - Default values

fbcc6a7

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Update versions.md & CMakeLists

efe7deb

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Revision - Headers & versions.md

ac0f539

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

Refs #20291: Adjust payload_pool test

8a9eede

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

cferreiragonz force-pushed the poc/gather-send-implementation branch from 394adb8 to 8a9eede Compare June 13, 2024 05:58

MiguelCompany self-requested a review June 13, 2024 05:59

MiguelCompany requested changes Jun 13, 2024

View reviewed changes

test/blackbox/common/DDSBlackboxTestsBasic.cpp Outdated Show resolved Hide resolved

Refs #20291: Test comment

4cf3f20

Signed-off-by: cferreiragonz <carlosferreira@eprosima.com>

MiguelCompany approved these changes Jun 13, 2024

View reviewed changes

MiguelCompany merged commit 371286c into master Jun 13, 2024
3 checks passed

MiguelCompany deleted the poc/gather-send-implementation branch June 13, 2024 12:23

elianalf mentioned this pull request Jun 14, 2024

Fix compilation warning #4945

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[20291] Gather-send implementation #4537

[20291] Gather-send implementation #4537

cferreiragonz commented Mar 8, 2024 •

edited by MiguelCompany

Loading

MiguelCompany left a comment

MiguelCompany commented Jun 13, 2024

[20291] Gather-send implementation #4537

[20291] Gather-send implementation #4537

Conversation

cferreiragonz commented Mar 8, 2024 • edited by MiguelCompany Loading

Description

Contributor Checklist

Reviewer Checklist

MiguelCompany left a comment

Choose a reason for hiding this comment

MiguelCompany commented Jun 13, 2024

cferreiragonz commented Mar 8, 2024 •

edited by MiguelCompany

Loading