Skip to content

Commit

Permalink
Documentation update
Browse files Browse the repository at this point in the history
Signed-off-by: Eugenio Collado <eugeniocollado@eprosima.com>
  • Loading branch information
EugenioCollado committed Dec 19, 2024
1 parent 9e25ba0 commit 52a4c85
Show file tree
Hide file tree
Showing 5 changed files with 85 additions and 27 deletions.
2 changes: 1 addition & 1 deletion ddsrecorder/test/blackbox/both/ResourceLimitsTest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -519,7 +519,7 @@ TEST_F(ResourceLimitsTest, sql_max_size)
}

/**
* @brief Test that the DDS Recorder's applies file-rotation to its output after reaching the max-size.
* @brief Test that the DDS Recorder's applies log-rotation to its output after reaching the max-size.
*
* In this test, the DDS Recorder's output is configured to have a max-size of 30KiB and a max-file-size of 7.5KiB.
* The DDS Recorder should create 3 output files, each of them with a size between 7KiB and 8KiB. Then, after
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,6 @@ ResourceLimitsConfiguration::ResourceLimitsConfiguration(
EPROSIMA_LOG_ERROR(YAML_READER_CONFIGURATION, "NOT VALID VALUE | SIZE TOLERANCE " << RECORDER_RESOURCE_LIMITS_SIZE_TOLERANCE_TAG << " must be greater than the minimum value accepted. Defaulting to (Mb): " << resource_limits_struct.size_tolerance_ / (1024 * 1024));
else
resource_limits_struct.size_tolerance_ = eprosima::utils::to_bytes(size_tolerance_str);
std::cout << "resource_limits_struct.size_tolerance_: " << resource_limits_struct.size_tolerance_ << std::endl;
}
}

Expand Down
86 changes: 71 additions & 15 deletions docs/rst/recording/usage/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -484,19 +484,56 @@ The supported compression options are:
Resource Limits
"""""""""""""""

The ``resource-limits`` tag allows users to limit the size of the *DDS Recorder's* output.
The ``max-file-size`` tag specifies the maximum size of each output file and the ``max-size`` tag specifies the maximum aggregate size of all output files.
If the ``max-size`` is higher than the ``max-file-size``, the |ddsrecorder| will create multiple files with a maximum size of ``max-file-size``.
By default, however, the ``max-file-size`` is unlimited (``0B``) and the ``max-size`` is the same as the ``max-file-size``; that is, by default the |ddsrecorder| creates a single file of unlimited size.
The ``safety-margin`` tag allows users to set a safety margin (in bytes) used when estimating the size of MCAP files.
By default, the safety margin is set to ``0``.
The ``resource-limits`` tag allows users to control the size of the *DDS Recorder's* output by setting limits on disk usage. This configuration allows distinct limits for the MCAP and SQL outputs while maintaining a shared safety margin to ensure stable memory usage.

- **``max-file-size``**: Specifies the maximum size of each output file. Applicable only to the MCAP recorder, as the SQL recorder uses a single database file.
- **``max-size``**: Specifies the maximum aggregate size of all output files. For the SQL recorder, this defines the maximum size of the database file. For the MCAP recorder, this determines the total size of all generated files.

### Safety Margin
The **``safety-margin``** property is shared between the SQL and MCAP outputs and is configured in the ``output`` section. This parameter reserves a buffer of free disk space, ensuring that at least ``safety-margin`` bytes remain available to prevent the system from running out of memory. This applies regardless of whether one or both recorders are enabled.
By default, the safety margin is set to ``10MB``.

### MCAP Recorder Behavior
If the ``max-size`` is greater than the ``max-file-size``, the |ddsrecorder| will create multiple files, each with a size up to the value of ``max-file-size``, until the total size reaches ``max-size``.

### SQL Recorder Behavior
For the SQL recorder:
- The database is always stored in a single file.
- **Both ``max-file-size`` and ``max-size`` control the same parameter, i.e., total size of the database**. This is why setting just one of them is sufficient as the other will be automatically set to the same value. If both are set to different values, an error will be returned.

### Default Behavior
By default:
- ``max-file-size`` is unlimited (``0B``).
- ``max-size`` is equal to ``max-file-size``, which means the |ddsrecorder| creates a single output file of unlimited size.

### Resource Limits Configuration Rules
The relation between ``max-size`` and ``safety-margin`` introduces resource limits that dictate memory usage. The behavior depends on the enabled recorders:

#### A. If only one recorder is enabled:
1. **No resource limits set**: The recorder will use all available disk space.
2. **Resource limits set**: Ensure the available space is sufficient to meet the specified limits.

#### B. If both recorders are enabled:
1. **No resource limits set**: Available disk space is divided equally between the recorders.
2. **One recorder with resource limits set**: The other recorder will use the remaining disk space.
3. **Both recorders with resource limits set**: Ensure the combined limits do not exceed the available disk space, returning an error otherwise.

### Size Margin
The **``size-margin``** property is an optional parameter that stablish the margin of error for the size of the output files.


.. warning::

If the ``max-file-size`` or the ``max-size`` are set to a value lower than the available space in the disk, the |ddsrecorder| will replace them with the available space in the disk.
If the ``max-file-size`` or the ``max-size`` are set to a value higher than the available space in the disk (counting for the safety-margin), an error will be returned.

To keep the |ddsrecorder| recording after reaching the ``max-size``, users can set the ``log-rotation`` tag to ``true``.
Enabling ``log-rotation`` allows the |ddsrecorder| to overwrite old files to free space for new ones.

### MCAP Log-Rotation Behavior
When the mcap ``log-rotation`` is enabled, the |ddsrecorder| will remove the oldest file whenever ``max-size`` is reached.

To keep the |ddsrecorder| recording after reaching the ``max-size``, users can set the ``file-rotation`` tag to ``true``.
Enabling ``file-rotation`` allows the |ddsrecorder| to overwrite old files to free space for new ones.
### SQL Log-Rotation Behavior
When the SQL ``log-rotation`` is enabled, the |ddsrecorder| will remove the oldest entries of the database whenever ``max-size`` is reached.

.. note::

Expand All @@ -511,11 +548,24 @@ Enabling ``file-rotation`` allows the |ddsrecorder| to overwrite old files to fr

.. code-block:: yaml
recorder:
output:
safety-margin: "1GB"

mcap:
enable: true
resource-limits:
max-file-size: 250KB
max-size: 2MiB
file-rotation: true
safety-margin: 10KB
max-file-size: 25MB
max-size: 200MB
log-rotation: true
size-margin: 2MB

sql:
enable: true
resource-limits:
max-size: 20MiB
log-rotation: true
size-margin: 1MB

.. _recorder_usage_configuration_sql:

Expand Down Expand Up @@ -861,6 +911,7 @@ A complete example of all the configurations described on this page can be found
path: "."
timestamp-format: "%Y-%m-%d_%H-%M-%S_%Z"
local-timestamp: false
safety-margin: 10GB
mcap:
enable: true
Expand All @@ -869,8 +920,8 @@ A complete example of all the configurations described on this page can be found
resource-limits:
max-file-size: 250KB
max-size: 2MiB
file-rotation: true
safety-margin: 10KB
log-rotation: true
size-tolerance: 10KB
compression:
algorithm: lz4
Expand All @@ -881,6 +932,11 @@ A complete example of all the configurations described on this page can be found
enable: false
data-format: "json"
resource-limits:
max-size: 2MiB
log-rotation: true
size-tolerance: 10KB
remote-controller:
enable: true
domain: 10
Expand Down
8 changes: 4 additions & 4 deletions resources/configurations/recorder/complete_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ recorder:
path: "."
timestamp-format: "%Y-%m-%d_%H-%M-%S_%Z"
local-timestamp: false
safety-margin: "1GB"

buffer-size: 50
event-window: 60
Expand All @@ -49,16 +50,15 @@ recorder:
resource-limits:
max-file-size: "100KB"
max-size: "300KB"
file-rotation: false
safety-margin: "1GB"
log-rotation: false
size-margin: "1MB"

sql:
enable: true
resource-limits:
max-size: "2MB"
log-rotation: false
safety-margin: "1GB"
size-margin: "1MB"


remote-controller:
Expand Down
15 changes: 9 additions & 6 deletions resources/configurations/recorder/minimal_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,20 @@ recorder:
output:
filename: "output"
path: "."
safety-margin: "1GB"

mcap:
enable: false
enable: true
resource-limits:
max-file-size: "100KB"
max-size: "100KB"
max-file-size: "20MB"
max-size: "100MB"
size-tolerance: "2MB"
log-rotation: true

sql:
enable: true
resource-limits:
max-size: "400KB"
max-size: "40MB"
size-tolerance: "4MB"
log-rotation: true
safety-margin: "1GB"

0 comments on commit 52a4c85

Please sign in to comment.