fix(llmobs): move listener hooks to enable instead of on init

[Yun-Kim]

Follow up on #11781 to fix a weird duplicate span writing issue with the new listener hook logic.

Since we were registering these hooks on LLMObs.__init__() which also happens at startup (as we create a default LLMObs() instance) as well as on LLMObs.enable(), we were double registering these hooks, and the default LLMObsSpanWriter was still saved and called each time the tracer finished a span. A symptom of this issue is that if a user was to manually enable agentless mode, they would see noisy logs indicating a failure to send spans to the agent proxy endpoint (which is the default writer mode) even though they also submitted spans to the agentless endpoint succesfully.

This fix resolves the issue by moving the hook registering to LLMObs.enable(), and adding corresponding logic to deregister the hooks on _stop_service(). This way we should only ever have one set of hooks registered per process.

This fix also creates an _on_span_start() callback instead of directly using _do_annotations() as the on span start hook. We currently have _do_annotations() as the only span start preprocessing logic, but this should be more conducive to future span start preprocessing steps that we might want to add.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

releasenotes/notes/fix-llmobs-default-writer-hooks-5e456c2f7dfd4381.yaml  @DataDog/apm-python
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

[pr-commenter]

Benchmarks

Benchmark execution time: 2025-01-10 16:40:49

Comparing candidate commit f0f6c03 in PR branch yunkim/llmobs-fix-hooks with baseline commit 04ee68f in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 394 metrics, 2 unstable metrics.

[Kyle-Verhoog]

ty for the fix, just one comment