feat(anta): Limit concurrency #680

carl-baillargeon · 2024-05-16T05:04:18Z

Description

This PR improves the test runner by introducing a generator-based approach for managing test coroutines and setting a configurable limit on the number of concurrent tests.

Instead of loading all test coroutines into a list, the runner now uses a generator to yield tests. This approach prevents memory overload and improves performance when dealing with a large number of tests.
A limit on the number of concurrent tests is introduced to avoid overwhelming the runner. This limit is configurable with an environement variable (hidden).

Implementation:
The generator yields test coroutines, ensuring that only a limited number of tests are scheduled and run concurrently.
Upon reaching the concurrency limit, the runner waits for some tests to complete before scheduling new ones from the generator.

Fixes: #713, #832

Checklist:

Update FAQ to reference Scaling ANTA documentation
Update Scaling ANTA documentation
Add section on JSON catalogs vs YAML (faster)
Need to retest on digital twin @dlobato
Support None timeouts
Confirm if PoolTimeout default should be set to None
Update benchmark + unit tests
ci: add codspeed to benchmark ANTA #826
Confirm that the number of open file descriptors never exceed device * max_connections (100 by default)

sonarqubecloud · 2024-06-12T16:43:30Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-07-04T14:03:49Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

anta/runner.py

github-actions · 2024-08-28T18:50:12Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

sonarqubecloud · 2024-08-28T19:07:52Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-11T13:47:41Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-10-02T15:05:25Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

codspeed-hq · 2024-10-02T15:15:13Z

CodSpeed Performance Report

Merging #680 will not alter performance

_{Comparing carl-baillargeon:refactor/runner_limit (17b244a) with main (bdc0e68)}

Summary

✅ 22 untouched benchmarks
🆕 2 new benchmarks

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
🆕	`test_setup_tests[1-device]`	N/A	15.7 ms	N/A
🆕	`test_setup_tests[2-devices]`	N/A	15.8 ms	N/A

github-actions · 2024-10-10T18:49:29Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-11-04T21:15:42Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

for more information, see https://pre-commit.ci

anta/runner.py

anta/settings.py

anta/cli/utils.py

mtache · 2025-01-21T10:50:50Z

mkdocs.yml

    - Developing ANTA tests: advanced_usages/custom-tests.md
    - ANTA as a Python Library: advanced_usages/as-python-lib.md
+    - Caching in ANTA: advanced_usages/caching.md
+    - Scaling ANTA: advanced_usages/scaling.md


Can we add a Settings section that render the BaseSettings models in anta.settings?

mtache · 2025-01-21T11:14:04Z

anta/settings.py

+
+    On POSIX systems, this value is used to set the soft limit for the current process.
+    The value cannot exceed the system's hard limit.
+    """


Missing ANTA_NOFILE attribute in the doctsring.

mtache · 2025-01-21T11:14:32Z

anta/settings.py

+
+
+class MaxConcurrencySettings(BaseSettings):
+    """Environment variable for configuring the maximum number of concurrent tests in the event loop."""


Missing ANTA_MAX_CONCURRENCY attribute in the docstring

mtache · 2025-01-21T11:15:10Z

anta/settings.py

+class HttpxResourceLimitsSettings(BaseSettings):
+    """Environment variables for configuring the underlying HTTPX client resource limits.
+
+    The limits are set using the following environment variables:


Use attribute section to have a table in mkdocs?

mtache · 2025-01-21T11:28:59Z

anta/_runner.py

+        if self.max_concurrency <= 0:
+            msg = "Concurrency limit must be greater than 0."
+            raise RuntimeError(msg)
+
+        # NOTE: The `aiter` built-in function is not available in Python 3.9
+        tests = generator.__aiter__()  # pylint: disable=unnecessary-dunder-call
+        tests_ended = False
+        tests_pending: set[Task[TestResult]] = set()
+
+        while tests_pending or not tests_ended:
+            # Add tests to the pending set until the limit is reached or no more tests are available
+            while len(tests_pending) < self.max_concurrency and not tests_ended:
+                try:
+                    # NOTE: The `anext` built-in function is not available in Python 3.9
+                    test = await tests.__anext__()  # pylint: disable=unnecessary-dunder-call
+                except StopAsyncIteration:  # noqa: PERF203
+                    tests_ended = True
+                    logger.debug("All tests have been added to the pending set.")
+                else:
+                    # Ensure the coroutine is scheduled to run and add it to the pending set
+                    tests_pending.add(asyncio.create_task(test))
+                    logger.debug("Added a test to the pending set: %s", test)
+
+            if len(tests_pending) >= self.max_concurrency:
+                logger.debug("Concurrency limit reached: %s tests running. Waiting for tests to complete.", self.max_concurrency)
+
+            if not tests_pending:
+                logger.debug("No pending tests and all tests have been processed. Exiting.")
+                return


Suggested change

if self.max_concurrency <= 0:

msg = "Concurrency limit must be greater than 0."

raise RuntimeError(msg)

# NOTE: The `aiter` built-in function is not available in Python 3.9

tests = generator.__aiter__() # pylint: disable=unnecessary-dunder-call

tests_ended = False

tests_pending: set[Task[TestResult]] = set()

while tests_pending or not tests_ended:

# Add tests to the pending set until the limit is reached or no more tests are available

while len(tests_pending) < self.max_concurrency and not tests_ended:

try:

# NOTE: The `anext` built-in function is not available in Python 3.9

test = await tests.__anext__() # pylint: disable=unnecessary-dunder-call

except StopAsyncIteration: # noqa: PERF203

tests_ended = True

logger.debug("All tests have been added to the pending set.")

else:

# Ensure the coroutine is scheduled to run and add it to the pending set

tests_pending.add(asyncio.create_task(test))

logger.debug("Added a test to the pending set: %s", test)

if len(tests_pending) >= self.max_concurrency:

logger.debug("Concurrency limit reached: %s tests running. Waiting for tests to complete.", self.max_concurrency)

if not tests_pending:

logger.debug("No pending tests and all tests have been processed. Exiting.")

return

if self.max_concurrency <= 0:

msg = "Concurrency limit must be greater than 0."

raise RuntimeError(msg)

tests_ended = False

tests_pending: set[Task[TestResult]] = set()

while tests_pending or not tests_ended:

async for test in generator:

# Add tests to the pending set until the limit is reached or no more tests are available

if len(tests_pending) < self.max_concurrency:

# Ensure the coroutine is scheduled to run and add it to the pending set

tests_pending.add(asyncio.create_task(test))

logger.debug("Added a test to the pending set: %s", test)

else:

logger.debug("Concurrency limit reached: %s tests running. Waiting for tests to complete.", self.max_concurrency)

break

if len(tests_pending) < self.max_concurrency and not tests_ended:

tests_ended = True

logger.debug("All tests have been added to the pending set.")

if not tests_pending:

logger.debug("No pending tests and all tests have been processed. Exiting.")

return

It looks equivalent right?

I didn't troubleshoot much but coroutines are not awaited properly with this code:

RuntimeWarning: Enable tracemalloc to get the object allocation traceback /home/cbaillar/git_projects/anta/anta/_runner.py:449: RuntimeWarning: coroutine 'VerifySSHStatus.test' was never awaited async for test in generator: RuntimeWarning: Enable tracemalloc to get the object allocation traceback /home/cbaillar/git_projects/anta/anta/_runner.py:449: RuntimeWarning: coroutine 'VerifyRoutingProtocolModel.test' was never awaited async for test in generator: RuntimeWarning: Enable tracemalloc to get the object allocation traceback /home/cbaillar/git_projects/anta/anta/_runner.py:449: RuntimeWarning: coroutine 'VerifyAcctConsoleMethods.test' was never awaited async for test in generator:

anta/device.py

anta/settings.py

mtache · 2025-01-21T15:15:22Z

tests/benchmark/test_anta.py

@@ -37,6 +37,7 @@ def test_anta_dry_run(

    results = session_results[request.node.callspec.id]

+    # TODO: Use AntaRunner in ANTA v2.0.0


Is this needed?
I think the anta.runner.main function uses the AntaRunner class already.

We can consider removing main() altogether and use AntaRunner everywhere in v2.0.0.

mtache · 2025-01-21T15:15:29Z

tests/benchmark/test_anta.py

@@ -69,6 +70,7 @@ def test_anta(

    results = session_results[request.node.callspec.id]

+    # TODO: Use AntaRunner in ANTA v2.0.0


I put this TODO everywhere we are using main().

mtache

Default value code could be simplified/cleaned.
Good idea to have this anta.settings module, we should use it for ANTA_DEBUG, in another PR ;)

github-actions · 2025-02-07T13:16:51Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2025-02-07T13:20:37Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

sonarqubecloud · 2025-02-07T20:17:04Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions bot assigned carl-baillargeon May 16, 2024

carl-baillargeon force-pushed the refactor/runner_limit branch from bf88158 to d7de654 Compare June 12, 2024 16:43

github-actions bot added the state: conflict label Jul 4, 2024

gmuloc reviewed Aug 1, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

gmuloc reviewed Aug 1, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

carl-baillargeon force-pushed the refactor/runner_limit branch from b623e90 to 4879120 Compare August 28, 2024 18:49

github-actions bot removed the state: conflict label Aug 28, 2024

github-actions bot added the state: conflict label Sep 11, 2024

mtache changed the title ~~refactor(anta): Refactor runner to use a generator with a limit~~ feat(anta): Limit concurrency Oct 2, 2024

feat(anta): limit the number of tests run concurrently

83206e8

mtache force-pushed the refactor/runner_limit branch from 68e259e to 83206e8 Compare October 2, 2024 15:05

github-actions bot removed the state: conflict label Oct 2, 2024

gmuloc and others added 2 commits October 7, 2024 09:53

Merge branch 'main' into refactor/runner_limit

d86f85c

Merge branch 'main' into refactor/runner_limit

6d59aa9

github-actions bot added the state: conflict label Oct 10, 2024

Merge branch 'main' into refactor/runner_limit

d2aa7c7

github-actions bot removed the state: conflict label Nov 4, 2024

gmuloc and others added 2 commits November 4, 2024 23:30

Refactor: Fix github merge conflict resolution

7ac0358

[pre-commit.ci] auto fixes from pre-commit.com hooks

96da6ef

for more information, see https://pre-commit.ci

gmuloc reviewed Nov 4, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

test: Temporarily disable one benchmark

f2f3c6a

carl-baillargeon requested review from mtache and dlobato January 19, 2025 19:20