chore(ci_visibility): add quarantine support to pytest

[vitor-de-araujo]

This PR adds preliminary support for quarantining tests in pytest.

The API to query which tests are quarantined does not exist yet on the backend side, and the final form of that API is still to be defined, so the code dealing with the API has been moved to a separate PR (#11770). Currently, we can mark tests as quarantined by manually adding the test.quarantine.is_quarantined tag to the test with a pytest decorator:

@pytest.mark.dd_tags(**{"test.quarantine.is_quarantined": True})
def test_fail_quarantined():
    assert False

For testing purposes, the environment variables _DD_TEST_FORCE_ENABLE_QUARANTINE and _DD_TEST_FORCE_ENABLE_ATR have been added to enable quarantine and ATR without depending on the backend.

The test reporting looks like below. Errors and logs for quarantined tests are not printed.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-12-20 12:17:07

Comparing candidate commit e419523 in PR branch vitor-de-araujo/SDTEST-1255/quarantine with baseline commit 20b2b03 in branch main.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 393 metrics, 2 unstable metrics.

scenario:iast_aspects-ospathdirname_aspect

• 🟩 execution_time [-402.878ns; -334.100ns] or [-9.979%; -8.276%]

[github-actions]

CODEOWNERS have been resolved as:

tests/ci_visibility/test_quarantine.py                                  @DataDog/ci-app-libraries
tests/contrib/pytest/test_pytest_quarantine.py                          @DataDog/ci-app-libraries
ddtrace/contrib/pytest/_atr_utils.py                                    @DataDog/ci-app-libraries
ddtrace/contrib/pytest/_plugin_v2.py                                    @DataDog/ci-app-libraries
ddtrace/contrib/pytest/_retry_utils.py                                  @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/_api_client.py                           @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_base.py                             @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_session.py                          @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_suite.py                            @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_test.py                             @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/constants.py                             @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/recorder.py                              @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/telemetry/events.py                      @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/telemetry/git.py                         @DataDog/ci-app-libraries
ddtrace/internal/test_visibility/api.py                                 @DataDog/ci-app-libraries
tests/ci_visibility/api_client/_util.py                                 @DataDog/ci-app-libraries
tests/ci_visibility/test_encoder.py                                     @DataDog/ci-app-libraries
tests/contrib/pytest/test_pytest_atr.py                                 @DataDog/ci-app-libraries
tests/contrib/pytest/test_pytest_efd.py                                 @DataDog/ci-app-libraries

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: vitor-de-araujo/SDTEST-1255/quarantine
Commit report: e419523
Test service: dd-trace-py

✅ 0 Failed, 922 Passed, 546 Skipped, 16m 53.61s Total duration (55m 44.02s time saved)