From 1811db145d431ae57170bdd417bec5487ede9ac1 Mon Sep 17 00:00:00 2001 From: Nick Ripley Date: Fri, 24 Jan 2025 16:35:37 -0500 Subject: [PATCH 01/13] chore(ci): increase profiling native test timeout (#12078) The profiling native tests have a 5 minute timeout. The test themselves typically finish in 1-2 minutes. But we see a pretty wide variance in how long the "Install llvm 19" step takes. Sometimes it takes 20 seconds and sometimes it takes over 3 minutes, making us time out. Increase the timeout just a bit to make the tests less flaky while we figure out a better way (caching? pre-building an image with llvm 19?) --- .github/workflows/profiling-native.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/profiling-native.yml b/.github/workflows/profiling-native.yml index 280d586d36e..09dd262b933 100644 --- a/.github/workflows/profiling-native.yml +++ b/.github/workflows/profiling-native.yml @@ -14,7 +14,7 @@ on: jobs: test: runs-on: ${{ matrix.os }} - timeout-minutes: 5 + timeout-minutes: 7 strategy: fail-fast: false matrix: From 67e1c46e0044be304843eb1403bc1f9b7c370000 Mon Sep 17 00:00:00 2001 From: Munir Abdinur Date: Sat, 25 Jan 2025 11:13:20 -0500 Subject: [PATCH 02/13] ci(appsec): fix ddwaf circular import (#12013) Fix the following CI failures: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-py/-/jobs/774636553 Reproduction: ``` from ddtrace.appsec._metrics import _set_waf_init_metric ``` - `ddtrace/appsec/_metrics` defines [_set_waf_init_metric](https://github.com/DataDog/dd-trace-py/blob/130b69b367e22311fa5fea7e3cc0910396e968c4/ddtrace/appsec/_metrics.py#L48-L49) - `ddtrace/appsec/_metrics` imports [ddtrace.appsec._proccessors](https://github.com/DataDog/dd-trace-py/blob/130b69b367e22311fa5fea7e3cc0910396e968c4/ddtrace/appsec/_metrics.py#L4-L5) - `ddtrace.appsec._proccessors` imports [_set_waf_init_metric](https://github.com/DataDog/dd-trace-py/blob/130b69b367e22311fa5fea7e3cc0910396e968c4/ddtrace/appsec/_processor.py#L443-L444) - Boom: Circular import The fix here is to avoid importing `ddwaf` from `ddtrace.appsec._processor` in `ddtrace.appsec._metrics`. Instead we can import ddwaf directly from `ddtrace.appsec._ddwaf`. This issue does not occur if the `ddtrace.appsec._processor` module is imported before `ddtrace.appsec._metrics`. This is why we do not see this error in most of our appsec tests. However the order of imports is not guaranteed by the slots check. This explains the flakiness in this job. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/appsec/_metrics.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ddtrace/appsec/_metrics.py b/ddtrace/appsec/_metrics.py index 75c329b50e1..ccab9d9455a 100644 --- a/ddtrace/appsec/_metrics.py +++ b/ddtrace/appsec/_metrics.py @@ -1,7 +1,7 @@ from ddtrace.appsec import _asm_request_context from ddtrace.appsec import _constants +import ddtrace.appsec._ddwaf as ddwaf from ddtrace.appsec._deduplications import deduplication -from ddtrace.appsec._processor import ddwaf from ddtrace.internal import telemetry from ddtrace.internal.logger import get_logger from ddtrace.internal.telemetry.constants import TELEMETRY_LOG_LEVEL From 75179ef6425d0318efb1f6c8aa31e072bc620c61 Mon Sep 17 00:00:00 2001 From: lievan <42917263+lievan@users.noreply.github.com> Date: Mon, 27 Jan 2025 00:32:47 -0500 Subject: [PATCH 03/13] fix(llmobs): fix token extraction for chat completion streams (#12070) Fixes token chunk extraction to account for the `choices` field in a chunk being an empty list #### Before ``` Error generating LLMObs span event for span , likely due to malformed span Traceback (most recent call last): File "/XXXXX/ddtrace/contrib/internal/openai/utils.py", line 118, in __aiter__ await self._extract_token_chunk(chunk) File "/XXXXX/ddtrace/contrib/internal/openai/utils.py", line 157, in _extract_token_chunk choice = getattr(chunk, "choices", [None])[0] IndexError: list index out of range ``` #### After Traced succesfully image ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: lievan Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> --- ddtrace/contrib/internal/openai/utils.py | 10 ++++++++-- .../notes/fix-token-extraction-0133808742374ef4.yaml | 4 ++++ 2 files changed, 12 insertions(+), 2 deletions(-) create mode 100644 releasenotes/notes/fix-token-extraction-0133808742374ef4.yaml diff --git a/ddtrace/contrib/internal/openai/utils.py b/ddtrace/contrib/internal/openai/utils.py index f5dfc10efef..0217b1e61d2 100644 --- a/ddtrace/contrib/internal/openai/utils.py +++ b/ddtrace/contrib/internal/openai/utils.py @@ -89,7 +89,10 @@ def _extract_token_chunk(self, chunk): """Attempt to extract the token chunk (last chunk in the stream) from the streamed response.""" if not self._dd_span._get_ctx_item("_dd.auto_extract_token_chunk"): return - choice = getattr(chunk, "choices", [None])[0] + choices = getattr(chunk, "choices") + if not choices: + return + choice = choices[0] if not getattr(choice, "finish_reason", None): # Only the second-last chunk in the stream with token usage enabled will have finish_reason set return @@ -152,7 +155,10 @@ async def _extract_token_chunk(self, chunk): """Attempt to extract the token chunk (last chunk in the stream) from the streamed response.""" if not self._dd_span._get_ctx_item("_dd.auto_extract_token_chunk"): return - choice = getattr(chunk, "choices", [None])[0] + choices = getattr(chunk, "choices") + if not choices: + return + choice = choices[0] if not getattr(choice, "finish_reason", None): return try: diff --git a/releasenotes/notes/fix-token-extraction-0133808742374ef4.yaml b/releasenotes/notes/fix-token-extraction-0133808742374ef4.yaml new file mode 100644 index 00000000000..cc8c1aa127b --- /dev/null +++ b/releasenotes/notes/fix-token-extraction-0133808742374ef4.yaml @@ -0,0 +1,4 @@ +--- +fixes: + - | + LLM Observability: This fix resolves an issue where extracting token metadata from openai streamed chat completion token chunks caused an IndexError. From b9228864ef7dc44d84f061316191a763d636972b Mon Sep 17 00:00:00 2001 From: Taegyun Kim Date: Mon, 27 Jan 2025 13:58:24 -0500 Subject: [PATCH 04/13] fix(ci, profiling): build native macos arm64 wheel (#12029) --- .github/workflows/build_python_3.yml | 5 ++- .../profiling/cmake/AnalysisFunc.cmake | 1 + .../profiling/crashtracker/CMakeLists.txt | 33 +++++++++---------- .../datadog/profiling/ddup/CMakeLists.txt | 12 +++---- .../datadog/profiling/stack_v2/CMakeLists.txt | 10 +++--- .../profiling/stack_v2/test/CMakeLists.txt | 5 +++ pyproject.toml | 6 ---- .../notes/ci-mac-build-37f7e55c8289de61.yaml | 6 ++++ 8 files changed, 43 insertions(+), 35 deletions(-) create mode 100644 releasenotes/notes/ci-mac-build-37f7e55c8289de61.yaml diff --git a/.github/workflows/build_python_3.yml b/.github/workflows/build_python_3.yml index a3d6066dc0c..a20779fbe81 100644 --- a/.github/workflows/build_python_3.yml +++ b/.github/workflows/build_python_3.yml @@ -35,7 +35,8 @@ jobs: cibuildwheel --print-build-identifiers --platform linux --arch x86_64,i686 | jq -cR '{only: ., os: "ubuntu-latest"}' \ && cibuildwheel --print-build-identifiers --platform linux --arch aarch64 | jq -cR '{only: ., os: "arm-4core-linux"}' \ && cibuildwheel --print-build-identifiers --platform windows --arch AMD64,x86 | grep -v 313 | jq -cR '{only: ., os: "windows-latest"}' \ - && cibuildwheel --print-build-identifiers --platform macos --arch x86_64,universal2 | jq -cR '{only: ., os: "macos-13"}' + && cibuildwheel --print-build-identifiers --platform macos --arch x86_64 | jq -cR '{only: ., os: "macos-13"}' \ + && cibuildwheel --print-build-identifiers --platform macos --arch arm64 | jq -cR '{only: ., os: "macos-latest"}' } | jq -sc ) echo $MATRIX_INCLUDE @@ -112,6 +113,7 @@ jobs: choco install -y 7zip && 7z d -r "{wheel}" *.c *.cpp *.cc *.h *.hpp *.pyx && move "{wheel}" "{dest_dir}" + CIBW_TEST_COMMAND: "python {project}/tests/smoke_test.py" # DEV: Uncomment to debug MacOS # CIBW_BUILD_VERBOSITY_MACOS: 3 @@ -152,6 +154,7 @@ jobs: choco install -y 7zip && 7z d -r "{wheel}" *.c *.cpp *.cc *.h *.hpp *.pyx && move "{wheel}" "{dest_dir}" + CIBW_TEST_COMMAND: "python {project}/tests/smoke_test.py" # DEV: Uncomment to debug MacOS # CIBW_BUILD_VERBOSITY_MACOS: 3 diff --git a/ddtrace/internal/datadog/profiling/cmake/AnalysisFunc.cmake b/ddtrace/internal/datadog/profiling/cmake/AnalysisFunc.cmake index 2495a84ed29..567ba15208f 100644 --- a/ddtrace/internal/datadog/profiling/cmake/AnalysisFunc.cmake +++ b/ddtrace/internal/datadog/profiling/cmake/AnalysisFunc.cmake @@ -32,6 +32,7 @@ function(add_ddup_config target) if(CMAKE_SYSTEM_NAME STREQUAL "Darwin") # macOS-specific linker options target_link_options(${target} PRIVATE "$<$:-Wl,-dead_strip>") + target_link_options(${target} PRIVATE -ldl -undefined dynamic_lookup) else() # Linux/ELF-based linker options target_link_options( diff --git a/ddtrace/internal/datadog/profiling/crashtracker/CMakeLists.txt b/ddtrace/internal/datadog/profiling/crashtracker/CMakeLists.txt index 8165613c07d..5bf8ee6ad80 100644 --- a/ddtrace/internal/datadog/profiling/crashtracker/CMakeLists.txt +++ b/ddtrace/internal/datadog/profiling/crashtracker/CMakeLists.txt @@ -58,16 +58,15 @@ set_target_properties(${EXTENSION_NAME} PROPERTIES SUFFIX "") # RPATH is needed for sofile discovery at runtime, since Python packages are not installed in the system path. This is # typical. -set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN/..") +if(APPLE) + set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "@loader_path/..") +elseif(UNIX) + set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN/..") +endif() target_include_directories(${EXTENSION_NAME} PRIVATE ../dd_wrapper/include ${Datadog_INCLUDE_DIRS} ${Python3_INCLUDE_DIRS}) -if(Python3_LIBRARIES) - target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper ${Python3_LIBRARIES}) -else() - # for manylinux builds - target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper) -endif() +target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper) # Set the output directory for the built library if(LIB_INSTALL_DIR) @@ -87,19 +86,19 @@ if(NOT CRASHTRACKER_EXE_TARGET_NAME) message(FATAL_ERROR "CRASHTRACKER_EXE_TARGET_NAME not set") endif() -set_target_properties(crashtracker_exe PROPERTIES INSTALL_RPATH "$ORIGIN/.." OUTPUT_NAME - ${CRASHTRACKER_EXE_TARGET_NAME}) +if(APPLE) + set_target_properties(crashtracker_exe PROPERTIES INSTALL_RPATH "@loader_path/.." OUTPUT_NAME + ${CRASHTRACKER_EXE_TARGET_NAME}) +elseif(UNIX) + set_target_properties(crashtracker_exe PROPERTIES INSTALL_RPATH "$ORIGIN/.." OUTPUT_NAME + ${CRASHTRACKER_EXE_TARGET_NAME}) -# To let crashtracker find Python library at runtime -set_target_properties(crashtracker_exe PROPERTIES INSTALL_RPATH_USE_LINK_PATH TRUE) - -if(Python3_LIBRARIES) - target_link_libraries(crashtracker_exe PRIVATE dd_wrapper ${Python3_LIBRARIES}) -else() - # for manylinux builds - target_link_libraries(crashtracker_exe PRIVATE dd_wrapper) + # To let crashtracker find Python library at runtime + set_target_properties(crashtracker_exe PROPERTIES INSTALL_RPATH_USE_LINK_PATH TRUE) endif() +target_link_libraries(crashtracker_exe PRIVATE dd_wrapper) + # See the dd_wrapper CMakeLists.txt for a more detailed explanation of why we do what we do. if(INPLACE_LIB_INSTALL_DIR) set(LIB_INSTALL_DIR "${INPLACE_LIB_INSTALL_DIR}") diff --git a/ddtrace/internal/datadog/profiling/ddup/CMakeLists.txt b/ddtrace/internal/datadog/profiling/ddup/CMakeLists.txt index 12ceab6fcb6..d4faa8704bb 100644 --- a/ddtrace/internal/datadog/profiling/ddup/CMakeLists.txt +++ b/ddtrace/internal/datadog/profiling/ddup/CMakeLists.txt @@ -61,15 +61,15 @@ set_target_properties(${EXTENSION_NAME} PROPERTIES SUFFIX "") # RPATH is needed for sofile discovery at runtime, since Python packages are not installed in the system path. This is # typical. -set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN/..") +if(APPLE) + set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "@loader_path/..") +elseif(UNIX) + set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN/..") +endif() target_include_directories(${EXTENSION_NAME} PRIVATE ../dd_wrapper/include ${Datadog_INCLUDE_DIRS} ${Python3_INCLUDE_DIRS}) -if(Python3_LIBRARIES) - target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper ${Python3_LIBRARIES}) -else() - target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper) -endif() +target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper) # Set the output directory for the built library if(LIB_INSTALL_DIR) diff --git a/ddtrace/internal/datadog/profiling/stack_v2/CMakeLists.txt b/ddtrace/internal/datadog/profiling/stack_v2/CMakeLists.txt index 77952e09d41..39bbdc42648 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/CMakeLists.txt +++ b/ddtrace/internal/datadog/profiling/stack_v2/CMakeLists.txt @@ -95,14 +95,14 @@ set_target_properties(${EXTENSION_NAME} PROPERTIES SUFFIX "") # RPATH is needed for sofile discovery at runtime, since Python packages are not installed in the system path. This is # typical. -set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN/..") +if(APPLE) + set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "@loader_path/..") +elseif(UNIX) + set_target_properties(${EXTENSION_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN/..") +endif() target_link_libraries(${EXTENSION_NAME} PRIVATE dd_wrapper Threads::Threads) -if(Python3_LIBRARIES) - target_link_libraries(${EXTENSION_NAME} PRIVATE ${Python3_LIBRARIES}) -endif() - # Set the output directory for the built library if(LIB_INSTALL_DIR) install( diff --git a/ddtrace/internal/datadog/profiling/stack_v2/test/CMakeLists.txt b/ddtrace/internal/datadog/profiling/stack_v2/test/CMakeLists.txt index 423f927d8f1..6cf4b2fe7a4 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/test/CMakeLists.txt +++ b/ddtrace/internal/datadog/profiling/stack_v2/test/CMakeLists.txt @@ -35,6 +35,11 @@ function(dd_wrapper_add_test name) target_include_directories(${name} PRIVATE ../include) # this has to refer to the stack_v2 extension name to properly link against target_link_libraries(${name} PRIVATE gmock gtest_main ${EXTENSION_NAME}) + + if(Python3_LIBRARIES) + target_link_libraries(${name} PRIVATE ${Python3_LIBRARIES}) + endif() + set_target_properties(${name} PROPERTIES INSTALL_RPATH "$ORIGIN/../stack_v2") add_ddup_config(${name}) diff --git a/pyproject.toml b/pyproject.toml index a959a80931f..0f2523d5fea 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -200,12 +200,6 @@ exclude_dirs = [ "ddtrace/sourcecode/_utils.py", ] -[tool.cibuildwheel] -build = ["cp27-*", "cp35-*", "cp36-*", "cp37-*", "cp38-*", "cp39-*", "cp310-*", "cp311-*"] -test-command = ["python {project}/tests/smoke_test.py"] -# Skip trying to test arm64 builds on Intel Macs -test-skip = "*-macosx_universal2:arm64" - [tool.ruff] exclude = [ ".riot", diff --git a/releasenotes/notes/ci-mac-build-37f7e55c8289de61.yaml b/releasenotes/notes/ci-mac-build-37f7e55c8289de61.yaml new file mode 100644 index 00000000000..5249b038c6e --- /dev/null +++ b/releasenotes/notes/ci-mac-build-37f7e55c8289de61.yaml @@ -0,0 +1,6 @@ +--- +fixes: + - | + profiling: fixes an issue on arm64 macOS where the profiler was not able + to find native extension modules to enable the following features: + ``DD_PROFILING_TIMELINE_ENABLED`` and ``DD_PROFILING_STACK_V2_ENABLED``. From b83292bef87fe69cc206de3267963dec3e8ab2c5 Mon Sep 17 00:00:00 2001 From: erikayasuda <153395705+erikayasuda@users.noreply.github.com> Date: Mon, 27 Jan 2025 21:44:13 +0100 Subject: [PATCH 05/13] ci: add GHA job to check if `main` can be merged to 3.x branches (#12093) Creates a GHA job that checks if the `main` branch can be cleanly merged to a 3.x-staging feature branch. This is to avoid scenarios where `main` has changes that conflict with `3.x-staging`, so we can resolve those before merging anything. This is mainly for the 3.x migration effort. We can clean this job up afterwards. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- .github/workflows/check_safe_main_merge.yml | 27 +++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 .github/workflows/check_safe_main_merge.yml diff --git a/.github/workflows/check_safe_main_merge.yml b/.github/workflows/check_safe_main_merge.yml new file mode 100644 index 00000000000..50a516ee4ac --- /dev/null +++ b/.github/workflows/check_safe_main_merge.yml @@ -0,0 +1,27 @@ +name: Check for Safe main Merge + +on: + pull_request: + branches: + - '3.x-staging' + +jobs: + check-merge: + runs-on: ubuntu-latest + steps: + # Step 1: Checkout the repository + - name: Checkout repository + uses: actions/checkout@v4 + + # Step 2: Fetch the main branch + - name: Fetch main branch + run: git fetch origin main + + # Step 3: Attempt to merge + - name: Check merge conflicts + run: | + git merge --no-commit --no-ff origin/main || exit 1 + # Step 4: Clean up the merge (optional) + - name: Abort merge + if: failure() + run: git merge --abort From aaedab9bcd69363043aaf47602288f96faf5fcfe Mon Sep 17 00:00:00 2001 From: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> Date: Mon, 27 Jan 2025 16:23:45 -0500 Subject: [PATCH 06/13] chore: update changelog for version 2.19.2 (#12088) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Nicole Cybul --- CHANGELOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a29299f8987..6c72665472f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,21 @@ Changelogs for versions not listed here can be found at https://github.com/DataD --- +## 2.19.2 +### Bug Fixes + +- Tracing + - celery: Fixes an issue where `celery.apply` spans from Celery prerun got closed too soon leading to span tags being missing. + - openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors. + - openai: Ensures the OpenAI integration is compatible with Python versions 3.12 and 3.13. + - vertexai: Resolves an issue with `chat.send_message()` where the content keyword argument was not parsed correctly. +- LLM Observability + - This fix resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors. +- Lib-Injection + - Fixes incorrect telemetry data payload format. + +--- + ## 2.19.1 ### Bug Fixes From e3045a19a20e0348f040aa589ef3c2517db74618 Mon Sep 17 00:00:00 2001 From: Nick Ripley Date: Mon, 27 Jan 2025 16:30:29 -0500 Subject: [PATCH 07/13] fix(profiling): fix SystemError when collecting memory profiler events (#12075) We added locking to the memory profiler to address crashes. These locks are mostly "try" locks, meaning we bail out if we can't acquire them right away. This was done defensively to mitigate the possibility of deadlock until we fully understood why the locks are needed and could guarantee their correctness. But as a result of using try locks, the `iter_events` function in particular can fail if the memory profiler lock is contended when it tries to collect profiling events. The function then returns NULL, leading to SystemError exceptions because we don't set an error. Even if we set an error, returning NULL isn't the right thing to do. It'll basically mean we wait until the next profile iteration, still accumulating events in the same buffer, and try again to upload the events. So we're going to get multiple iteration's worth of events. The right thing to do is take the lock unconditionally in `iter_events`. We can allocate the new tracker outside the memory allocation profiler lock so that we don't need to worry about reentrancy/deadlock issues if we start profiling that allocation. Then, the only thing we do under the lock is swap out the global tracker, so it's safe to take the lock unconditionally. Fixes #11831 TODO - regression test? --- ddtrace/profiling/collector/_memalloc.c | 22 +++++++++++++------ ...loc-iter-events-null-780fd50bbebbf616.yaml | 4 ++++ 2 files changed, 19 insertions(+), 7 deletions(-) create mode 100644 releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml diff --git a/ddtrace/profiling/collector/_memalloc.c b/ddtrace/profiling/collector/_memalloc.c index 1f2b87e0433..1e1b9dbf52e 100644 --- a/ddtrace/profiling/collector/_memalloc.c +++ b/ddtrace/profiling/collector/_memalloc.c @@ -394,20 +394,28 @@ iterevents_new(PyTypeObject* type, PyObject* Py_UNUSED(args), PyObject* Py_UNUSE } IterEventsState* iestate = (IterEventsState*)type->tp_alloc(type, 0); - if (!iestate) + if (!iestate) { + PyErr_SetString(PyExc_RuntimeError, "failed to allocate IterEventsState"); return NULL; + } memalloc_assert_gil(); - /* reset the current traceback list */ - if (memlock_trylock(&g_memalloc_lock)) { - iestate->alloc_tracker = global_alloc_tracker; - global_alloc_tracker = alloc_tracker_new(); - memlock_unlock(&g_memalloc_lock); - } else { + /* Reset the current traceback list. Do this outside lock so we can track it, + * and avoid reentrancy/deadlock problems, if we start tracking the raw + * allocator domain */ + alloc_tracker_t* tracker = alloc_tracker_new(); + if (!tracker) { + PyErr_SetString(PyExc_RuntimeError, "failed to allocate new allocation tracker"); Py_TYPE(iestate)->tp_free(iestate); return NULL; } + + memlock_lock(&g_memalloc_lock); + iestate->alloc_tracker = global_alloc_tracker; + global_alloc_tracker = tracker; + memlock_unlock(&g_memalloc_lock); + iestate->seq_index = 0; PyObject* iter_and_count = PyTuple_New(3); diff --git a/releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml b/releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml new file mode 100644 index 00000000000..52a43cbd2a1 --- /dev/null +++ b/releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml @@ -0,0 +1,4 @@ +--- +fixes: + - | + profiling: fix SystemError from the memory profiler returning NULL when collecting events From 55767a78063ed17de0e954a6db4b9ff64b762f1d Mon Sep 17 00:00:00 2001 From: William Conti <58711692+wconti27@users.noreply.github.com> Date: Tue, 28 Jan 2025 08:44:03 -0500 Subject: [PATCH 08/13] chore(tracing): refactor web server integrations to use the core module (#12035) ## Motivation Refactors all web server integrations still using `tracer.trace` to instead use `core.context_with_data`. This is in preparation for supporting AWS API Gateway to ensure all web servers share the same code path. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/_trace/trace_handlers.py | 56 +++++++++- .../contrib/internal/aiohttp/middlewares.py | 105 ++++++++---------- ddtrace/contrib/internal/bottle/trace.py | 61 +++++----- .../contrib/internal/cherrypy/middleware.py | 53 +++++---- ddtrace/contrib/internal/falcon/middleware.py | 61 ++++------ ddtrace/contrib/internal/molten/patch.py | 52 ++++----- ddtrace/contrib/internal/pyramid/trace.py | 72 ++++++------ ddtrace/contrib/internal/sanic/patch.py | 61 +++++----- ...b.cherrypy.test_middleware.test_child.json | 1 + ...b.cherrypy.test_middleware.test_error.json | 1 + ...b.cherrypy.test_middleware.test_fatal.json | 1 + ...cherrypy.test_middleware.test_success.json | 1 + ...ver.test_multiple_requests_sanic_http.json | 2 + ...multiple_requests_sanic_http_pre_21.9.json | 2 + ...c.test_sanic_server.test_sanic_errors.json | 2 + ...nic_server.test_sanic_errors_pre_21.9.json | 2 + 16 files changed, 290 insertions(+), 243 deletions(-) diff --git a/ddtrace/_trace/trace_handlers.py b/ddtrace/_trace/trace_handlers.py index dab3f743146..3ebb28f33c8 100644 --- a/ddtrace/_trace/trace_handlers.py +++ b/ddtrace/_trace/trace_handlers.py @@ -109,11 +109,14 @@ def _get_parameters_for_new_span_directly_from_context(ctx: core.ExecutionContex def _start_span(ctx: core.ExecutionContext, call_trace: bool = True, **kwargs) -> "Span": span_kwargs = _get_parameters_for_new_span_directly_from_context(ctx) call_trace = ctx.get_item("call_trace", call_trace) - tracer = (ctx.get_item("middleware") or ctx["pin"]).tracer + tracer = ctx.get_item("tracer") or (ctx.get_item("middleware") or ctx["pin"]).tracer distributed_headers_config = ctx.get_item("distributed_headers_config") if distributed_headers_config: trace_utils.activate_distributed_headers( - tracer, int_config=distributed_headers_config, request_headers=ctx["distributed_headers"] + tracer, + int_config=distributed_headers_config, + request_headers=ctx["distributed_headers"], + override=ctx.get_item("distributed_headers_config_override"), ) distributed_context = ctx.get_item("distributed_context") if distributed_context and not call_trace: @@ -126,6 +129,42 @@ def _start_span(ctx: core.ExecutionContext, call_trace: bool = True, **kwargs) - return span +def _set_web_frameworks_tags(ctx, span, int_config): + span.set_tag_str(COMPONENT, int_config.integration_name) + span.set_tag_str(SPAN_KIND, SpanKind.SERVER) + span.set_tag(_SPAN_MEASURED_KEY) + + analytics_enabled = ctx.get_item("analytics_enabled") + analytics_sample_rate = ctx.get_item("analytics_sample_rate", True) + + # Configure trace search sample rate + if (config._analytics_enabled and analytics_enabled is not False) or analytics_enabled is True: + span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, analytics_sample_rate) + + +def _on_web_framework_start_request(ctx, int_config): + request_span = ctx.get_item("req_span") + _set_web_frameworks_tags(ctx, request_span, int_config) + + +def _on_web_framework_finish_request( + span, int_config, method, url, status_code, query, req_headers, res_headers, route, finish +): + trace_utils.set_http_meta( + span=span, + integration_config=int_config, + method=method, + url=url, + status_code=status_code, + query=query, + request_headers=req_headers, + response_headers=res_headers, + route=route, + ) + if finish: + span.finish() + + def _on_traced_request_context_started_flask(ctx): current_span = ctx["pin"].tracer.current_span() if not ctx["pin"].enabled or not current_span: @@ -761,6 +800,10 @@ def listen(): core.on("azure.functions.request_call_modifier", _on_azure_functions_request_span_modifier) core.on("azure.functions.start_response", _on_azure_functions_start_response) + # web frameworks general handlers + core.on("web.request.start", _on_web_framework_start_request) + core.on("web.request.finish", _on_web_framework_finish_request) + core.on("test_visibility.enable", _on_test_visibility_enable) core.on("test_visibility.disable", _on_test_visibility_disable) core.on("test_visibility.is_enabled", _on_test_visibility_is_enabled, "is_enabled") @@ -769,6 +812,14 @@ def listen(): core.on("rq.queue.enqueue_job", _propagate_context) for context_name in ( + # web frameworks + "aiohttp.request", + "bottle.request", + "cherrypy.request", + "falcon.request", + "molten.request", + "pyramid.request", + "sanic.request", "flask.call", "flask.jsonify", "flask.render_template", @@ -779,6 +830,7 @@ def listen(): "django.template.render", "django.process_exception", "django.func.wrapped", + # non web frameworks "botocore.instrumented_api_call", "botocore.instrumented_lib_function", "botocore.patched_kinesis_api_call", diff --git a/ddtrace/contrib/internal/aiohttp/middlewares.py b/ddtrace/contrib/internal/aiohttp/middlewares.py index ddb2d35fbc6..63a60734d3b 100644 --- a/ddtrace/contrib/internal/aiohttp/middlewares.py +++ b/ddtrace/contrib/internal/aiohttp/middlewares.py @@ -2,15 +2,10 @@ from aiohttp.web_urldispatcher import SystemRoute from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils from ddtrace.contrib.asyncio import context_provider -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes from ddtrace.ext import http -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import schematize_url_operation from ddtrace.internal.schema.span_attribute_schema import SpanDirection @@ -35,47 +30,42 @@ async def attach_context(request): # application configs tracer = app[CONFIG_KEY]["tracer"] service = app[CONFIG_KEY]["service"] - distributed_tracing = app[CONFIG_KEY]["distributed_tracing_enabled"] - # Create a new context based on the propagated information. - trace_utils.activate_distributed_headers( - tracer, - int_config=config.aiohttp, - request_headers=request.headers, - override=distributed_tracing, - ) - - # trace the handler - request_span = tracer.trace( - schematize_url_operation("aiohttp.request", protocol="http", direction=SpanDirection.INBOUND), - service=service, - span_type=SpanTypes.WEB, - ) - request_span.set_tag(_SPAN_MEASURED_KEY) - - request_span.set_tag_str(COMPONENT, config.aiohttp.integration_name) - - # set span.kind tag equal to type of request - request_span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - # Configure trace search sample rate # DEV: aiohttp is special case maintains separate configuration from config api analytics_enabled = app[CONFIG_KEY]["analytics_enabled"] - if (config._analytics_enabled and analytics_enabled is not False) or analytics_enabled is True: - request_span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, app[CONFIG_KEY].get("analytics_sample_rate", True)) - - # attach the context and the root span to the request; the Context - # may be freely used by the application code - request[REQUEST_CONTEXT_KEY] = request_span.context - request[REQUEST_SPAN_KEY] = request_span - request[REQUEST_CONFIG_KEY] = app[CONFIG_KEY] - try: - response = await handler(request) - if isinstance(response, web.StreamResponse): - request.task.add_done_callback(lambda _: finish_request_span(request, response)) - return response - except Exception: - request_span.set_traceback() - raise + # Create a new context based on the propagated information. + + with core.context_with_data( + "aiohttp.request", + span_name=schematize_url_operation("aiohttp.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, + service=service, + tags={}, + tracer=tracer, + distributed_headers=request.headers, + distributed_headers_config=config.aiohttp, + distributed_headers_config_override=app[CONFIG_KEY]["distributed_tracing_enabled"], + headers_case_sensitive=True, + analytics_enabled=analytics_enabled, + analytics_sample_rate=app[CONFIG_KEY].get("analytics_sample_rate", True), + ) as ctx: + req_span = ctx.span + + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.aiohttp)) + + # attach the context and the root span to the request; the Context + # may be freely used by the application code + request[REQUEST_CONTEXT_KEY] = req_span.context + request[REQUEST_SPAN_KEY] = req_span + request[REQUEST_CONFIG_KEY] = app[CONFIG_KEY] + try: + response = await handler(request) + if isinstance(response, web.StreamResponse): + request.task.add_done_callback(lambda _: finish_request_span(request, response)) + return response + except Exception: + req_span.set_traceback() + raise return attach_context @@ -122,19 +112,22 @@ def finish_request_span(request, response): # SystemRoute objects exist to throw HTTP errors and have no path route = aiohttp_route.resource.canonical - trace_utils.set_http_meta( - request_span, - config.aiohttp, - method=request.method, - url=str(request.url), # DEV: request.url is a yarl's URL object - status_code=response.status, - request_headers=request.headers, - response_headers=response.headers, - route=route, + core.dispatch( + "web.request.finish", + ( + request_span, + config.aiohttp, + request.method, + str(request.url), # DEV: request.url is a yarl's URL object + response.status, + None, # query arg = None + request.headers, + response.headers, + route, + True, + ), ) - request_span.finish() - async def on_prepare(request, response): """ diff --git a/ddtrace/contrib/internal/bottle/trace.py b/ddtrace/contrib/internal/bottle/trace.py index 3aabb4ccc81..58778a36ee1 100644 --- a/ddtrace/contrib/internal/bottle/trace.py +++ b/ddtrace/contrib/internal/bottle/trace.py @@ -5,13 +5,8 @@ import ddtrace from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import schematize_url_operation from ddtrace.internal.schema.span_attribute_schema import SpanDirection from ddtrace.internal.utils.formats import asbool @@ -42,24 +37,21 @@ def wrapped(*args, **kwargs): resource = "{} {}".format(request.method, route.rule) - trace_utils.activate_distributed_headers( - self.tracer, int_config=config.bottle, request_headers=request.headers - ) - - with self.tracer.trace( - schematize_url_operation("bottle.request", protocol="http", direction=SpanDirection.INBOUND), + with core.context_with_data( + "bottle.request", + span_name=schematize_url_operation("bottle.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, service=self.service, resource=resource, - span_type=SpanTypes.WEB, - ) as s: - s.set_tag_str(COMPONENT, config.bottle.integration_name) - - # set span.kind to the type of request being performed - s.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - s.set_tag(_SPAN_MEASURED_KEY) - # set analytics sample rate with global config enabled - s.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, config.bottle.get_analytics_sample_rate(use_global_config=True)) + tags={}, + tracer=self.tracer, + distributed_headers=request.headers, + distributed_headers_config=config.bottle, + headers_case_sensitive=True, + analytics_sample_rate=config.bottle.get_analytics_sample_rate(use_global_config=True), + ) as ctx, ctx.span as req_span: + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.bottle)) code = None result = None @@ -91,16 +83,21 @@ def wrapped(*args, **kwargs): method = request.method url = request.urlparts._replace(query="").geturl() full_route = "/".join([request.script_name.rstrip("/"), route.rule.lstrip("/")]) - trace_utils.set_http_meta( - s, - config.bottle, - method=method, - url=url, - status_code=response_code, - query=request.query_string, - request_headers=request.headers, - response_headers=response.headers, - route=full_route, + + core.dispatch( + "web.request.finish", + ( + req_span, + config.bottle, + method, + url, + response_code, + request.query_string, + request.headers, + response.headers, + full_route, + False, + ), ) return wrapped diff --git a/ddtrace/contrib/internal/cherrypy/middleware.py b/ddtrace/contrib/internal/cherrypy/middleware.py index 909a043175f..70f43059905 100644 --- a/ddtrace/contrib/internal/cherrypy/middleware.py +++ b/ddtrace/contrib/internal/cherrypy/middleware.py @@ -11,12 +11,10 @@ from ddtrace.constants import ERROR_MSG from ddtrace.constants import ERROR_STACK from ddtrace.constants import ERROR_TYPE -from ddtrace.constants import SPAN_KIND from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes from ddtrace.internal import compat -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import SpanDirection from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation @@ -77,20 +75,23 @@ def _setup(self): cherrypy.request.hooks.attach("after_error_response", self._after_error_response, priority=5) def _on_start_resource(self): - trace_utils.activate_distributed_headers( - self._tracer, int_config=config.cherrypy, request_headers=cherrypy.request.headers - ) - - cherrypy.request._datadog_span = self._tracer.trace( - SPAN_NAME, - service=trace_utils.int_service(None, config.cherrypy, default="cherrypy"), + with core.context_with_data( + "cherrypy.request", + span_name=SPAN_NAME, span_type=SpanTypes.WEB, - ) + service=trace_utils.int_service(None, config.cherrypy, default="cherrypy"), + tags={}, + tracer=self._tracer, + distributed_headers=cherrypy.request.headers, + distributed_headers_config=config.cherrypy, + headers_case_sensitive=True, + ) as ctx: + req_span = ctx.span - cherrypy.request._datadog_span.set_tag_str(COMPONENT, config.cherrypy.integration_name) + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.cherrypy)) - # set span.kind to the type of request being performed - cherrypy.request._datadog_span.set_tag_str(SPAN_KIND, SpanKind.SERVER) + cherrypy.request._datadog_span = req_span def _after_error_response(self): span = getattr(cherrypy.request, "_datadog_span", None) @@ -135,18 +136,22 @@ def _close_span(self, span): url = compat.to_unicode(cherrypy.request.base + cherrypy.request.path_info) status_code, _, _ = valid_status(cherrypy.response.status) - trace_utils.set_http_meta( - span, - config.cherrypy, - method=cherrypy.request.method, - url=url, - status_code=status_code, - request_headers=cherrypy.request.headers, - response_headers=cherrypy.response.headers, + core.dispatch( + "web.request.finish", + ( + span, + config.cherrypy, + cherrypy.request.method, + url, + status_code, + None, + cherrypy.request.headers, + cherrypy.response.headers, + None, + True, + ), ) - span.finish() - # Clear our span just in case. cherrypy.request._datadog_span = None diff --git a/ddtrace/contrib/internal/falcon/middleware.py b/ddtrace/contrib/internal/falcon/middleware.py index b4ec5434777..513ce6cce39 100644 --- a/ddtrace/contrib/internal/falcon/middleware.py +++ b/ddtrace/contrib/internal/falcon/middleware.py @@ -1,14 +1,8 @@ import sys from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes -from ddtrace.ext import http as httpx -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import SpanDirection from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation @@ -27,26 +21,27 @@ def __init__(self, tracer, service=None, distributed_tracing=None): def process_request(self, req, resp): # Falcon uppercases all header names. headers = dict((k.lower(), v) for k, v in req.headers.items()) - trace_utils.activate_distributed_headers(self.tracer, int_config=config.falcon, request_headers=headers) - span = self.tracer.trace( - schematize_url_operation("falcon.request", protocol="http", direction=SpanDirection.INBOUND), - service=self.service, + with core.context_with_data( + "falcon.request", + span_name=schematize_url_operation("falcon.request", protocol="http", direction=SpanDirection.INBOUND), span_type=SpanTypes.WEB, - ) - span.set_tag_str(COMPONENT, config.falcon.integration_name) - - # set span.kind to the type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - span.set_tag(_SPAN_MEASURED_KEY) - - # set analytics sample rate with global config enabled - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, config.falcon.get_analytics_sample_rate(use_global_config=True)) - - trace_utils.set_http_meta( - span, config.falcon, method=req.method, url=req.url, query=req.query_string, request_headers=req.headers - ) + service=self.service, + tags={}, + tracer=self.tracer, + distributed_headers=headers, + distributed_headers_config=config.falcon, + headers_case_sensitive=True, + analytics_sample_rate=config.falcon.get_analytics_sample_rate(use_global_config=True), + ) as ctx: + req_span = ctx.span + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.falcon)) + + core.dispatch( + "web.request.finish", + (req_span, config.falcon, req.method, req.url, None, req.query_string, req.headers, None, None, False), + ) def process_resource(self, req, resp, resource, params): span = self.tracer.current_span() @@ -69,8 +64,7 @@ def process_response(self, req, resp, resource, req_succeeded=None): if resource is None: status = "404" span.resource = "%s 404" % req.method - span.set_tag(httpx.STATUS_CODE, status) - span.finish() + core.dispatch("web.request.finish", (span, config.falcon, None, None, status, None, None, None, None, True)) return err_type = sys.exc_info()[0] @@ -87,20 +81,13 @@ def process_response(self, req, resp, resource, req_succeeded=None): route = req.root_path or "" + req.uri_template - trace_utils.set_http_meta( - span, - config.falcon, - status_code=status, - response_headers=resp._headers, - route=route, - ) - # Emit span hook for this response # DEV: Emit before closing so they can overwrite `span.resource` if they want config.falcon.hooks.emit("request", span, req, resp) - # Close the span - span.finish() + core.dispatch( + "web.request.finish", (span, config.falcon, None, None, status, None, None, resp._headers, route, True) + ) def _is_404(err_type): diff --git a/ddtrace/contrib/internal/molten/patch.py b/ddtrace/contrib/internal/molten/patch.py index 38fa949243c..dfec47eb17d 100644 --- a/ddtrace/contrib/internal/molten/patch.py +++ b/ddtrace/contrib/internal/molten/patch.py @@ -5,15 +5,11 @@ from wrapt import wrap_function_wrapper as _w from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND from ddtrace.contrib import trace_utils from ddtrace.contrib.internal.trace_utils import unwrap as _u -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes +from ddtrace.internal import core from ddtrace.internal.compat import urlencode -from ddtrace.internal.constants import COMPONENT from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation from ddtrace.internal.schema.span_attribute_schema import SpanDirection @@ -89,25 +85,21 @@ def patch_app_call(wrapped, instance, args, kwargs): request = molten.http.Request.from_environ(environ) resource = func_name(wrapped) - # request.headers is type Iterable[Tuple[str, str]] - trace_utils.activate_distributed_headers( - pin.tracer, int_config=config.molten, request_headers=dict(request.headers) - ) - - with pin.tracer.trace( - schematize_url_operation("molten.request", protocol="http", direction=SpanDirection.INBOUND), + with core.context_with_data( + "molten.request", + span_name=schematize_url_operation("molten.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, service=trace_utils.int_service(pin, config.molten), resource=resource, - span_type=SpanTypes.WEB, - ) as span: - span.set_tag_str(COMPONENT, config.molten.integration_name) - - # set span.kind tag equal to type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - span.set_tag(_SPAN_MEASURED_KEY) - # set analytics sample rate with global config enabled - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, config.molten.get_analytics_sample_rate(use_global_config=True)) + tags={}, + tracer=pin.tracer, + distributed_headers=dict(request.headers), # request.headers is type Iterable[Tuple[str, str]] + distributed_headers_config=config.molten, + headers_case_sensitive=True, + analytics_sample_rate=config.molten.get_analytics_sample_rate(use_global_config=True), + ) as ctx, ctx.span as req_span: + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.molten)) @wrapt.function_wrapper def _w_start_response(wrapped, instance, args, kwargs): @@ -125,11 +117,13 @@ def _w_start_response(wrapped, instance, args, kwargs): except ValueError: pass - if not span.get_tag(MOLTEN_ROUTE): + if not req_span.get_tag(MOLTEN_ROUTE): # if route never resolve, update root resource - span.resource = "{} {}".format(request.method, code) + req_span.resource = "{} {}".format(request.method, code) - trace_utils.set_http_meta(span, config.molten, status_code=code) + core.dispatch( + "web.request.finish", (req_span, config.molten, None, None, code, None, None, None, None, False) + ) return wrapped(*args, **kwargs) @@ -143,11 +137,13 @@ def _w_start_response(wrapped, instance, args, kwargs): request.path, ) query = urlencode(dict(request.params)) - trace_utils.set_http_meta( - span, config.molten, method=request.method, url=url, query=query, request_headers=request.headers + + core.dispatch( + "web.request.finish", + (req_span, config.molten, request.method, url, None, query, request.headers, None, None, False), ) - span.set_tag_str("molten.version", molten.__version__) + req_span.set_tag_str("molten.version", molten.__version__) return wrapped(environ, start_response, **kwargs) diff --git a/ddtrace/contrib/internal/pyramid/trace.py b/ddtrace/contrib/internal/pyramid/trace.py index 9942c673d4e..9b221bd3f09 100644 --- a/ddtrace/contrib/internal/pyramid/trace.py +++ b/ddtrace/contrib/internal/pyramid/trace.py @@ -6,12 +6,8 @@ # project import ddtrace from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes +from ddtrace.internal import core from ddtrace.internal.constants import COMPONENT from ddtrace.internal.logger import get_logger from ddtrace.internal.schema import schematize_service_name @@ -67,29 +63,30 @@ def trace_tween_factory(handler, registry): service = settings.get(SETTINGS_SERVICE) or schematize_service_name("pyramid") tracer = settings.get(SETTINGS_TRACER) or ddtrace.tracer enabled = asbool(settings.get(SETTINGS_TRACE_ENABLED, tracer.enabled)) - distributed_tracing = asbool(settings.get(SETTINGS_DISTRIBUTED_TRACING, True)) + + # ensure distributed tracing within pyramid settings matches config + config.pyramid.distributed_tracing_enabled = asbool(settings.get(SETTINGS_DISTRIBUTED_TRACING, True)) if enabled: # make a request tracing function def trace_tween(request): - trace_utils.activate_distributed_headers( - tracer, int_config=config.pyramid, request_headers=request.headers, override=distributed_tracing - ) - - span_name = schematize_url_operation("pyramid.request", protocol="http", direction=SpanDirection.INBOUND) - with tracer.trace(span_name, service=service, resource="404", span_type=SpanTypes.WEB) as span: - span.set_tag_str(COMPONENT, config.pyramid.integration_name) - - # set span.kind to the type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - span.set_tag(_SPAN_MEASURED_KEY) - # Configure trace search sample rate + with core.context_with_data( + "pyramid.request", + span_name=schematize_url_operation("pyramid.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, + service=service, + resource="404", + tags={}, + tracer=tracer, + distributed_headers=request.headers, + distributed_headers_config=config.pyramid, + headers_case_sensitive=True, # DEV: pyramid is special case maintains separate configuration from config api - analytics_enabled = settings.get(SETTINGS_ANALYTICS_ENABLED) - - if (config._analytics_enabled and analytics_enabled is not False) or analytics_enabled is True: - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, settings.get(SETTINGS_ANALYTICS_SAMPLE_RATE, True)) + analytics_enabled=settings.get(SETTINGS_ANALYTICS_ENABLED), + analytics_sample_rate=settings.get(SETTINGS_ANALYTICS_SAMPLE_RATE, True), + ) as ctx, ctx.span as req_span: + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.pyramid)) setattr(request, DD_TRACER, tracer) # used to find the tracer in templates response = None @@ -110,8 +107,8 @@ def trace_tween(request): finally: # set request tags if request.matched_route: - span.resource = "{} {}".format(request.method, request.matched_route.name) - span.set_tag_str("pyramid.route.name", request.matched_route.name) + req_span.resource = "{} {}".format(request.method, request.matched_route.name) + req_span.set_tag_str("pyramid.route.name", request.matched_route.name) # set response tags if response: status = response.status_code @@ -119,17 +116,22 @@ def trace_tween(request): else: response_headers = None - trace_utils.set_http_meta( - span, - config.pyramid, - method=request.method, - url=request.path_url, - status_code=status, - query=request.query_string, - request_headers=request.headers, - response_headers=response_headers, - route=request.matched_route.pattern if request.matched_route else None, + core.dispatch( + "web.request.finish", + ( + req_span, + config.pyramid, + request.method, + request.path_url, + status, + request.query_string, + request.headers, + response_headers, + request.matched_route.pattern if request.matched_route else None, + False, + ), ) + return response return trace_tween diff --git a/ddtrace/contrib/internal/sanic/patch.py b/ddtrace/contrib/internal/sanic/patch.py index 8e53ed41dc8..1babf7d44a0 100644 --- a/ddtrace/contrib/internal/sanic/patch.py +++ b/ddtrace/contrib/internal/sanic/patch.py @@ -4,14 +4,10 @@ import wrapt from wrapt import wrap_function_wrapper as _w -import ddtrace from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import SPAN_KIND from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.logger import get_logger from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation @@ -47,7 +43,10 @@ def update_span(span, response): # and so use 500 status_code = getattr(response, "status", 500) response_headers = getattr(response, "headers", None) - trace_utils.set_http_meta(span, config.sanic, status_code=status_code, response_headers=response_headers) + + core.dispatch( + "web.request.finish", (span, config.sanic, None, None, status_code, None, None, response_headers, None, False) + ) def _wrap_response_callback(span, callback): @@ -200,31 +199,35 @@ def _create_sanic_request_span(request): headers = request.headers.copy() - trace_utils.activate_distributed_headers(ddtrace.tracer, int_config=config.sanic, request_headers=headers) - - span = pin.tracer.trace( - schematize_url_operation("sanic.request", protocol="http", direction=SpanDirection.INBOUND), + with core.context_with_data( + "sanic.request", + span_name=schematize_url_operation("sanic.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, service=trace_utils.int_service(None, config.sanic), resource=resource, - span_type=SpanTypes.WEB, - ) - span.set_tag_str(COMPONENT, config.sanic.integration_name) - - # set span.kind to the type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - sample_rate = config.sanic.get_analytics_sample_rate(use_global_config=True) - if sample_rate is not None: - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, sample_rate) - - method = request.method - url = "{scheme}://{host}{path}".format(scheme=request.scheme, host=request.host, path=request.path) - query_string = request.query_string - if isinstance(query_string, bytes): - query_string = query_string.decode() - trace_utils.set_http_meta(span, config.sanic, method=method, url=url, query=query_string, request_headers=headers) - - return span + tags={}, + pin=pin, + distributed_headers=headers, + distributed_headers_config=config.sanic, + headers_case_sensitive=True, + analytics_sample_rate=config.sanic.get_analytics_sample_rate(use_global_config=True), + ) as ctx: + req_span = ctx.span + + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.sanic)) + + method = request.method + url = "{scheme}://{host}{path}".format(scheme=request.scheme, host=request.host, path=request.path) + query_string = request.query_string + if isinstance(query_string, bytes): + query_string = query_string.decode() + + core.dispatch( + "web.request.finish", (req_span, config.sanic, method, url, None, query_string, headers, None, None, False) + ) + + return req_span async def sanic_http_lifecycle_handle(request): diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json index 1887282b06f..933eda3e420 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json index 49759fee3d3..58fde524828 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json @@ -25,6 +25,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json index 4153c8c2b18..a5a7c1ed14f 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json @@ -25,6 +25,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json index d1eb9fc5bee..ce13988add0 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json index 3803d07f360..66ca28d9606 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json @@ -23,6 +23,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -72,6 +73,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json index 4aac2721c02..aca376a974a 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -70,6 +71,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json index 435bbbc7b23..ab2aeaec920 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -58,6 +59,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json index c03813a43d6..ceabecd1ae7 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -54,6 +55,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, From 16d5280956bc259e77691536229824d3183a4181 Mon Sep 17 00:00:00 2001 From: Christophe Papazian <114495376+christophe-papazian@users.noreply.github.com> Date: Tue, 28 Jan 2025 15:48:22 +0100 Subject: [PATCH 09/13] ci(tracer): make serverless test unrequired (#12121) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- .gitlab/benchmarks.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitlab/benchmarks.yml b/.gitlab/benchmarks.yml index e922d315444..692a61ea93f 100644 --- a/.gitlab/benchmarks.yml +++ b/.gitlab/benchmarks.yml @@ -96,6 +96,7 @@ benchmark-serverless: tags: ["arch:amd64"] when: on_success needs: [ "benchmark-serverless-trigger" ] + allow_failure: true script: - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/serverless-tools.git ./serverless-tools && cd ./serverless-tools - ./ci/check_trigger_status.sh From 4f0bcb5a59378fb6015230f9f9ad04199fc380b5 Mon Sep 17 00:00:00 2001 From: Christophe Papazian <114495376+christophe-papazian@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:28:20 +0100 Subject: [PATCH 10/13] chore(asm): clean libddwaf loading (#12102) Depending of the timing, libddwaf loading process could create triggers that would create loops in our instrumentation. From what I investigated: - if loaded too early, it could have bad interactions with gevent. - if loaded too late, it could be self instrumented by the tracer, creating a loop, as ctypes is using Popen and subprocess. while keeping the late loading introduced by 2 previous PRs - https://github.com/DataDog/dd-trace-py/pull/11987 - https://github.com/DataDog/dd-trace-py/pull/12013 this PR introduced a mechanism to bypass tracer instrumentation during ctypes loading, to avoid a possible loop that would prevent the WAF to be loaded. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/appsec/_ddwaf/ddwaf_types.py | 3 +++ ddtrace/appsec/_processor.py | 26 +++++++++++--------- ddtrace/contrib/internal/subprocess/patch.py | 6 +++++ ddtrace/settings/asm.py | 1 + 4 files changed, 24 insertions(+), 12 deletions(-) diff --git a/ddtrace/appsec/_ddwaf/ddwaf_types.py b/ddtrace/appsec/_ddwaf/ddwaf_types.py index e998b6048f6..5942a2fa184 100644 --- a/ddtrace/appsec/_ddwaf/ddwaf_types.py +++ b/ddtrace/appsec/_ddwaf/ddwaf_types.py @@ -22,9 +22,12 @@ if system() == "Linux": try: + asm_config._bypass_instrumentation_for_waf = True ctypes.CDLL(ctypes.util.find_library("rt"), mode=ctypes.RTLD_GLOBAL) except Exception: # nosec pass + finally: + asm_config._bypass_instrumentation_for_waf = False ARCHI = machine().lower() diff --git a/ddtrace/appsec/_processor.py b/ddtrace/appsec/_processor.py index 6102ba1ded2..83b65584a66 100644 --- a/ddtrace/appsec/_processor.py +++ b/ddtrace/appsec/_processor.py @@ -4,6 +4,7 @@ from json.decoder import JSONDecodeError import os import os.path +from typing import TYPE_CHECKING from typing import Any from typing import Dict from typing import List @@ -11,6 +12,11 @@ from typing import Set from typing import Tuple from typing import Union + + +if TYPE_CHECKING: + import ddtrace.appsec._ddwaf as ddwaf + import weakref from ddtrace._trace.processor import SpanProcessor @@ -167,14 +173,17 @@ def __post_init__(self) -> None: def delayed_init(self) -> None: try: if self._rules is not None and not hasattr(self, "_ddwaf"): - self._ddwaf = ddwaf.DDWaf( + from ddtrace.appsec._ddwaf import DDWaf # noqa: E402 + import ddtrace.appsec._metrics as metrics # noqa: E402 + + self.metrics = metrics + self._ddwaf = DDWaf( self._rules, self.obfuscation_parameter_key_regexp, self.obfuscation_parameter_value_regexp ) - _set_waf_init_metric(self._ddwaf.info) + self.metrics._set_waf_init_metric(self._ddwaf.info) except Exception: # Partial of DDAS-0005-00 log.warning("[DDAS-0005-00] WAF initialization failed") - raise self._update_required() def _update_required(self): @@ -193,7 +202,7 @@ def _update_rules(self, new_rules: Dict[str, Any]) -> bool: if asm_config._asm_static_rule_file is not None: return result result = self._ddwaf.update_rules(new_rules) - _set_waf_updates_metric(self._ddwaf.info) + self.metrics._set_waf_updates_metric(self._ddwaf.info) self._update_required() return result @@ -241,7 +250,7 @@ def waf_callable(custom_data=None, **kwargs): return self._waf_action(span._local_root or span, ctx, custom_data, **kwargs) _asm_request_context.set_waf_callback(waf_callable) - _asm_request_context.add_context_callback(_set_waf_request_metrics) + _asm_request_context.add_context_callback(self.metrics._set_waf_request_metrics) if headers is not None: _asm_request_context.set_waf_address(SPAN_DATA_NAMES.REQUEST_HEADERS_NO_COOKIES, headers) _asm_request_context.set_waf_address( @@ -436,10 +445,3 @@ def on_span_finish(self, span: Span) -> None: del self._span_to_waf_ctx[s] except Exception: # nosec B110 pass - - -# load waf at the end only to avoid possible circular imports with gevent -import ddtrace.appsec._ddwaf as ddwaf # noqa: E402 -from ddtrace.appsec._metrics import _set_waf_init_metric # noqa: E402 -from ddtrace.appsec._metrics import _set_waf_request_metrics # noqa: E402 -from ddtrace.appsec._metrics import _set_waf_updates_metric # noqa: E402 diff --git a/ddtrace/contrib/internal/subprocess/patch.py b/ddtrace/contrib/internal/subprocess/patch.py index 80d05b107bb..2d66edd4737 100644 --- a/ddtrace/contrib/internal/subprocess/patch.py +++ b/ddtrace/contrib/internal/subprocess/patch.py @@ -327,6 +327,8 @@ def unpatch() -> None: @trace_utils.with_traced_module def _traced_ossystem(module, pin, wrapped, instance, args, kwargs): try: + if asm_config._bypass_instrumentation_for_waf: + return wrapped(*args, **kwargs) if isinstance(args[0], str): for callback in _STR_CALLBACKS.values(): callback(args[0]) @@ -393,6 +395,8 @@ def _traced_osspawn(module, pin, wrapped, instance, args, kwargs): @trace_utils.with_traced_module def _traced_subprocess_init(module, pin, wrapped, instance, args, kwargs): try: + if asm_config._bypass_instrumentation_for_waf: + return wrapped(*args, **kwargs) cmd_args = args[0] if len(args) else kwargs["args"] if isinstance(cmd_args, (list, tuple, str)): if kwargs.get("shell", False): @@ -425,6 +429,8 @@ def _traced_subprocess_init(module, pin, wrapped, instance, args, kwargs): @trace_utils.with_traced_module def _traced_subprocess_wait(module, pin, wrapped, instance, args, kwargs): try: + if asm_config._bypass_instrumentation_for_waf: + return wrapped(*args, **kwargs) binary = core.get_item("subprocess_popen_binary") with pin.tracer.trace(COMMANDS.SPAN_NAME, resource=binary, span_type=SpanTypes.SYSTEM) as span: diff --git a/ddtrace/settings/asm.py b/ddtrace/settings/asm.py index 16937399501..fc2c9aa13fb 100644 --- a/ddtrace/settings/asm.py +++ b/ddtrace/settings/asm.py @@ -245,6 +245,7 @@ class ASMConfig(Env): default=r"^[+-]?((0b[01]+)|(0x[0-9A-Fa-f]+)|(\d+\.?\d*(?:[Ee][+-]?\d+)?|\.\d+(?:[Ee][+-]" + r"?\d+)?)|(X\'[0-9A-Fa-f]+\')|(B\'[01]+\'))$", ) + _bypass_instrumentation_for_waf = False def __init__(self): super().__init__() From c4448ea4e5580cfa55684ef108a5dafafe1f6f75 Mon Sep 17 00:00:00 2001 From: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:11:42 -0500 Subject: [PATCH 11/13] fix(llmobs): propagate distributed headers via signal dispatching, not config (#12089) This PR makes a change to our shared distributed tracing header injection method to dispatch signals/events instead of relying on the global config settings, which is only modifiable via env vars. This fixes distributed tracing for users that might rely solely on the `LLMObs.enable()` setup config. Programmatic `LLMObs.enable()/disable()` calls do not set the global `config._llmobs_enabled` boolean setting, which is only controlled by the `DD_LLMOBS_ENABLED` env var. This was problematic for users that relied on manual `LLMObs.enable()` setup (i.e. no env vars) because our distributed tracing injection code only checks the global config to inject llmobs parent IDs into request headers. If users manually enabled LLMObs without any env vars, then this would not be reflected in the global config value and thus LLMObs parent IDs would never be injected into the request headers. We can't check directly if LLMObs is enabled in the http injection module because: 1. This would require us to import significant product-specific LLMObs-code into the shared http injector helper module which would impact non-LLMObs users' app performance 2. Circular imports in LLMObs which imports http injector logic to use in its own helpers Instead of doing our check based on the global `config._llmobs_enabled` setting, we now send a tracing event to our shared product listeners, and register a corresponding `LLMObs._inject_llmobs_context()` hook to be called for all inject() calls if LLMObs is enabled (we check the LLMObs instance, not the global config setting value). ~One risk and why I don't like changing global config settings is because this then implies that it is no longer global or tied to an env var (I want to push for env var configuration where possible over manual overriding/enabling). If a global enabled config can be toggled indiscriminately then this could open a can of worms for enabling/disabling logic in our LLMObs service, which isn't really designed to be toggled on/off multiple times in the app's lifespan. However if some users cannot rely on env vars, then I don't see any other solution that does not couple tracer internal code with LLMObs code which is a no-option.~ (UPDATE: we avoided this issue by using signal dispatching) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/llmobs/_llmobs.py | 8 ++- ddtrace/propagation/http.py | 7 +-- ...nable-updates-config-45379a7a30e2e0e3.yaml | 5 ++ tests/llmobs/test_llmobs_service.py | 1 - tests/llmobs/test_propagation.py | 54 ++++++++++++------- tests/tracer/test_propagation.py | 24 --------- 6 files changed, 49 insertions(+), 50 deletions(-) create mode 100644 releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml diff --git a/ddtrace/llmobs/_llmobs.py b/ddtrace/llmobs/_llmobs.py index 768f4bdb292..bc5ddabcf00 100644 --- a/ddtrace/llmobs/_llmobs.py +++ b/ddtrace/llmobs/_llmobs.py @@ -285,6 +285,7 @@ def _stop_service(self) -> None: # Remove listener hooks for span events core.reset_listeners("trace.span_start", self._on_span_start) core.reset_listeners("trace.span_finish", self._on_span_finish) + core.reset_listeners("http.span_inject", self._inject_llmobs_context) forksafe.unregister(self._child_after_fork) @@ -369,6 +370,7 @@ def enable( # Register hooks for span events core.on("trace.span_start", cls._instance._on_span_start) core.on("trace.span_finish", cls._instance._on_span_finish) + core.on("http.span_inject", cls._instance._inject_llmobs_context) atexit.register(cls.disable) telemetry_writer.product_activated(TELEMETRY_APM_PRODUCT.LLMOBS, True) @@ -1162,6 +1164,11 @@ def submit_evaluation( cls._instance._llmobs_eval_metric_writer.enqueue(evaluation_metric) + def _inject_llmobs_context(self, span_context: Context, request_headers: Dict[str, str]) -> None: + if self.enabled is False: + return + _inject_llmobs_parent_id(span_context) + @classmethod def inject_distributed_headers(cls, request_headers: Dict[str, str], span: Optional[Span] = None) -> Dict[str, str]: """Injects the span's distributed context into the given request headers.""" @@ -1179,7 +1186,6 @@ def inject_distributed_headers(cls, request_headers: Dict[str, str], span: Optio if span is None: log.warning("No span provided and no currently active span found.") return request_headers - _inject_llmobs_parent_id(span.context) HTTPPropagator.inject(span.context, request_headers) return request_headers diff --git a/ddtrace/propagation/http.py b/ddtrace/propagation/http.py index fdaf97410ad..381acabb1bc 100644 --- a/ddtrace/propagation/http.py +++ b/ddtrace/propagation/http.py @@ -28,6 +28,7 @@ from ddtrace._trace.span import _get_64_lowest_order_bits_as_int from ddtrace._trace.span import _MetaDictType from ddtrace.appsec._constants import APPSEC +from ddtrace.internal.core import dispatch from ddtrace.settings.asm import config as asm_config from ..constants import AUTO_KEEP @@ -1052,6 +1053,7 @@ def parent_call(): :param dict headers: HTTP headers to extend with tracing attributes. :param Span non_active_span: Only to be used if injecting a non-active span. """ + dispatch("http.span_inject", (span_context, headers)) if not config._propagation_style_inject: return if non_active_span is not None and non_active_span.context is not span_context: @@ -1089,11 +1091,6 @@ def parent_call(): for key in span_context._baggage: headers[_HTTP_BAGGAGE_PREFIX + key] = span_context._baggage[key] - if config._llmobs_enabled: - from ddtrace.llmobs._utils import _inject_llmobs_parent_id - - _inject_llmobs_parent_id(span_context) - if PROPAGATION_STYLE_DATADOG in config._propagation_style_inject: _DatadogMultiHeader._inject(span_context, headers) if PROPAGATION_STYLE_B3_MULTI in config._propagation_style_inject: diff --git a/releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml b/releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml new file mode 100644 index 00000000000..4bf312f4680 --- /dev/null +++ b/releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml @@ -0,0 +1,5 @@ +--- +fixes: + - | + LLM Observability: Resolves an issue where explicitly only using ``LLMObs.enable()`` to configure LLM Observability + without environment variables would not automatically propagate distributed tracing headers. diff --git a/tests/llmobs/test_llmobs_service.py b/tests/llmobs/test_llmobs_service.py index de428999147..744fde885e9 100644 --- a/tests/llmobs/test_llmobs_service.py +++ b/tests/llmobs/test_llmobs_service.py @@ -60,7 +60,6 @@ def test_service_enable_proxy_default(): assert llmobs_instance.tracer == dummy_tracer assert isinstance(llmobs_instance._llmobs_span_writer._clients[0], LLMObsProxiedEventClient) assert run_llmobs_trace_filter(dummy_tracer) is not None - llmobs_service.disable() diff --git a/tests/llmobs/test_propagation.py b/tests/llmobs/test_propagation.py index e3ab9c80d66..27421e6b12f 100644 --- a/tests/llmobs/test_propagation.py +++ b/tests/llmobs/test_propagation.py @@ -57,20 +57,24 @@ def test_propagate_correct_llmobs_parent_id_simple(run_python_code_in_subprocess """ code = """ import json +import mock -from ddtrace import tracer -from ddtrace.ext import SpanTypes +from ddtrace.internal.utils.http import Response +from ddtrace.llmobs import LLMObs from ddtrace.propagation.http import HTTPPropagator -with tracer.trace("LLMObs span", span_type=SpanTypes.LLM) as root_span: - with tracer.trace("Non-LLMObs span") as child_span: - headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} - HTTPPropagator.inject(child_span.context, headers) +with mock.patch( + "ddtrace.internal.writer.HTTPWriter._send_payload", return_value=Response(status=200, body="{}"), +): + LLMObs.enable(ml_app="test-app", api_key="", agentless_enabled=True) + with LLMObs.workflow("LLMObs span") as root_span: + with LLMObs._instance.tracer.trace("Non-LLMObs span") as child_span: + headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} + HTTPPropagator.inject(child_span.context, headers) print(json.dumps(headers)) """ env = os.environ.copy() - env["DD_LLMOBS_ENABLED"] = "1" env["DD_TRACE_ENABLED"] = "0" stdout, stderr, status, _ = run_python_code_in_subprocess(code=code, env=env) assert status == 0, (stdout, stderr) @@ -93,21 +97,33 @@ def test_propagate_llmobs_parent_id_complex(run_python_code_in_subprocess): """ code = """ import json +import mock -from ddtrace import tracer -from ddtrace.ext import SpanTypes +from ddtrace.internal.utils.http import Response +from ddtrace.llmobs import LLMObs from ddtrace.propagation.http import HTTPPropagator -with tracer.trace("LLMObs span", span_type=SpanTypes.LLM) as root_span: - with tracer.trace("Non-LLMObs span") as child_span: - headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} - HTTPPropagator.inject(child_span.context, headers) +with mock.patch( + "ddtrace.internal.writer.HTTPWriter._send_payload", return_value=Response(status=200, body="{}"), +): + from ddtrace import auto # simulate ddtrace-run startup to ensure env var configs also propagate + with LLMObs.workflow("LLMObs span") as root_span: + with LLMObs._instance.tracer.trace("Non-LLMObs span") as child_span: + headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} + HTTPPropagator.inject(child_span.context, headers) print(json.dumps(headers)) """ env = os.environ.copy() - env["DD_LLMOBS_ENABLED"] = "1" - env["DD_TRACE_ENABLED"] = "0" + env.update( + { + "DD_LLMOBS_ENABLED": "1", + "DD_TRACE_ENABLED": "0", + "DD_AGENTLESS_ENABLED": "1", + "DD_API_KEY": "", + "DD_LLMOBS_ML_APP": "test-app", + } + ) stdout, stderr, status, _ = run_python_code_in_subprocess(code=code, env=env) assert status == 0, (stdout, stderr) assert stderr == b"", (stdout, stderr) @@ -124,7 +140,7 @@ def test_propagate_llmobs_parent_id_complex(run_python_code_in_subprocess): def test_no_llmobs_parent_id_propagated_if_no_llmobs_spans(run_python_code_in_subprocess): - """Test that the correct LLMObs parent ID (None) is extracted from the headers in a simple distributed scenario. + """Test that the correct LLMObs parent ID ('undefined') is extracted from headers in a simple distributed scenario. Service A (subprocess) has spans, but none are LLMObs spans. Service B (outside subprocess) has a LLMObs span. Service B's span should have no LLMObs parent ID as there are no LLMObs spans from service A. @@ -132,17 +148,17 @@ def test_no_llmobs_parent_id_propagated_if_no_llmobs_spans(run_python_code_in_su code = """ import json -from ddtrace import tracer +from ddtrace.llmobs import LLMObs from ddtrace.propagation.http import HTTPPropagator -with tracer.trace("Non-LLMObs span") as root_span: +LLMObs.enable(ml_app="ml-app", agentless_enabled=True, api_key="") +with LLMObs._instance.tracer.trace("Non-LLMObs span") as root_span: headers = {} HTTPPropagator.inject(root_span.context, headers) print(json.dumps(headers)) """ env = os.environ.copy() - env["DD_LLMOBS_ENABLED"] = "1" env["DD_TRACE_ENABLED"] = "0" stdout, stderr, status, _ = run_python_code_in_subprocess(code=code, env=env) assert status == 0, (stdout, stderr) diff --git a/tests/tracer/test_propagation.py b/tests/tracer/test_propagation.py index 533e4974250..b073cd72e3b 100644 --- a/tests/tracer/test_propagation.py +++ b/tests/tracer/test_propagation.py @@ -4,7 +4,6 @@ import os import pickle -import mock import pytest import ddtrace @@ -3387,29 +3386,6 @@ def test_DD_TRACE_PROPAGATION_STYLE_INJECT_overrides_DD_TRACE_PROPAGATION_STYLE( assert result == expected_headers -def test_llmobs_enabled_injects_llmobs_parent_id(): - with override_global_config(dict(_llmobs_enabled=True)): - with mock.patch("ddtrace.llmobs._utils._inject_llmobs_parent_id") as mock_llmobs_inject: - context = Context(trace_id=1, span_id=2) - HTTPPropagator.inject(context, {}) - mock_llmobs_inject.assert_called_once_with(context) - - -def test_llmobs_disabled_does_not_inject_parent_id(): - with override_global_config(dict(_llmobs_enabled=False)): - with mock.patch("ddtrace.llmobs._utils._inject_llmobs_parent_id") as mock_llmobs_inject: - context = Context(trace_id=1, span_id=2) - HTTPPropagator.inject(context, {}) - mock_llmobs_inject.assert_not_called() - - -def test_llmobs_parent_id_not_injected_by_default(): - with mock.patch("ddtrace.llmobs._utils._inject_llmobs_parent_id") as mock_llmobs_inject: - context = Context(trace_id=1, span_id=2) - HTTPPropagator.inject(context, {}) - mock_llmobs_inject.assert_not_called() - - @pytest.mark.parametrize( "span_context,expected_headers", [ From cb41f8e77aa71bfffbb63f7ee1a9808e659189d2 Mon Sep 17 00:00:00 2001 From: Munir Abdinur Date: Wed, 29 Jan 2025 08:39:44 -0500 Subject: [PATCH 12/13] feat(provider): expose context provider in ddtrace.trace (#12135) ddtrace v3.0 is set to remove `ddtrace.providers` module from the public API. However we should still allow users to use their own ContextProviders. This PR ensures the BaseContextProvider remains in the public API and can be used in `tracer.configure(...)`. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- .github/CODEOWNERS | 2 ++ ddtrace/provider.py | 5 +++-- ddtrace/trace/__init__.py | 2 ++ docs/advanced_usage.rst | 2 +- .../feat-expose-base-context-provider-530ebec2225f6c8d.yaml | 5 +++++ 5 files changed, 13 insertions(+), 3 deletions(-) create mode 100644 releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 84e6c873642..b1162da2348 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -166,12 +166,14 @@ ddtrace/internal/remoteconfig @DataDog/remote-config @DataDog/apm-core-pyt tests/internal/remoteconfig @DataDog/remote-config @DataDog/apm-core-python # API SDK +ddtrace/trace/ @DataDog/apm-sdk-api-python ddtrace/_trace/ @DataDog/apm-sdk-api-python ddtrace/opentelemetry/ @DataDog/apm-sdk-api-python ddtrace/internal/opentelemetry @DataDog/apm-sdk-api-python ddtrace/opentracer/ @DataDog/apm-sdk-api-python ddtrace/propagation/ @DataDog/apm-sdk-api-python ddtrace/filters.py @DataDog/apm-sdk-api-python +ddtrace/provider.py @DataDog/apm-sdk-api-python ddtrace/pin.py @DataDog/apm-sdk-api-python ddtrace/sampler.py @DataDog/apm-sdk-api-python ddtrace/sampling_rule.py @DataDog/apm-sdk-api-python diff --git a/ddtrace/provider.py b/ddtrace/provider.py index 4a275ece8c8..7b9867de01a 100644 --- a/ddtrace/provider.py +++ b/ddtrace/provider.py @@ -7,7 +7,8 @@ deprecate( - "The context provider interface is deprecated.", - message="The trace context is an internal interface and will no longer be supported.", + "The context provider interface is deprecated", + message="Import BaseContextProvider from `ddtrace.trace` instead.", category=DDTraceDeprecationWarning, + removal_version="3.0.0", ) diff --git a/ddtrace/trace/__init__.py b/ddtrace/trace/__init__.py index f709310d589..8473c0b99f0 100644 --- a/ddtrace/trace/__init__.py +++ b/ddtrace/trace/__init__.py @@ -1,6 +1,7 @@ from ddtrace._trace.context import Context from ddtrace._trace.filters import TraceFilter from ddtrace._trace.pin import Pin +from ddtrace._trace.provider import BaseContextProvider from ddtrace._trace.span import Span from ddtrace._trace.tracer import Tracer @@ -15,4 +16,5 @@ "Tracer", "Span", "tracer", + "BaseContextProvider", ] diff --git a/docs/advanced_usage.rst b/docs/advanced_usage.rst index 05879acac37..392e1406fb7 100644 --- a/docs/advanced_usage.rst +++ b/docs/advanced_usage.rst @@ -221,7 +221,7 @@ context management. If there is a case where the default is insufficient then a custom context provider can be used. It must implement the -:class:`ddtrace.provider.BaseContextProvider` interface and can be configured +:class:`ddtrace.trace.BaseContextProvider` interface and can be configured with:: tracer.configure(context_provider=MyContextProvider) diff --git a/releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml b/releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml new file mode 100644 index 00000000000..010d2bf52bd --- /dev/null +++ b/releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml @@ -0,0 +1,5 @@ +--- +deprecations: + - | + tracing: Moves ``ddtrace.provider.BaseContextProvider`` to ``ddtrace.trace.BaseContextProvider``. + The ``ddtrace.provider`` module is deprecated and will be removed in v3.0.0. From af9098c4343831d4b8771fe5debd6f99d021eac6 Mon Sep 17 00:00:00 2001 From: Laplie Anderson Date: Thu, 30 Jan 2025 05:34:25 -0500 Subject: [PATCH 13/13] chore(ci): skip non-linux OCI package creation (#12036) The shared pipeline is introducing support for windows OCI packages. This PR ensure dd-trace-py doesn't generate nonsensical packages. Windows support can be added later. This was tested by using the WIP branch of one-pipeline ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Brett Langdon --- .gitlab/prepare-oci-package.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.gitlab/prepare-oci-package.sh b/.gitlab/prepare-oci-package.sh index 5958c31e731..7ee7b7d6e77 100755 --- a/.gitlab/prepare-oci-package.sh +++ b/.gitlab/prepare-oci-package.sh @@ -1,6 +1,11 @@ #!/bin/bash set -eo pipefail +if [ "$OS" != "linux" ]; then + echo "Only linux packages are supported. Exiting" + exit 0 +fi + if [ -n "$CI_COMMIT_TAG" ] && [ -z "$PYTHON_PACKAGE_VERSION" ]; then PYTHON_PACKAGE_VERSION=${CI_COMMIT_TAG##v} fi