chore(langchain): drop support for tracing `langchain.chains` #12172

sabrenner · 2025-01-30T16:47:26Z

This PR drops support for tracing langchain.chains in favor of only tracing RunnableSequence (LangChain Expression Language - LCEL) chains.

High LOC - this PR also re-introduces APM tracing tests for langchain, but removes all cassettes and snapshots related to previous langchain patching. We still need to refactor conftest a bit to be more modern, and re-introduce LLMObs tests. I'm fairly certain those ones are flaky, while the normal APM tests should be OK.

Known issue: This PR effectively removes all patching on the langchain package, and only patches langchain_core and community and partner libraries. However, to trigger patching, you must still:

import langchain

in order to trigger patching for LangChain. This will be fixed in a patch in ddtrace 3.x.

MLOB-2140

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

github-actions · 2025-01-30T16:47:56Z

CODEOWNERS have been resolved as:

releasenotes/notes/langchain-drop-langchain-base-chains-support-48d8290af6eb86e3.yaml  @DataDog/apm-python
tests/contrib/langchain/cassettes/openai_chat_completion_image_input_sync_generate.yaml  @DataDog/ml-observability
tests/contrib/langchain/test_langchain.py                               @DataDog/ml-observability
ddtrace/contrib/internal/langchain/patch.py                             @DataDog/ml-observability
tests/contrib/langchain/conftest.py                                     @DataDog/ml-observability
tests/contrib/langchain/test_langchain_patch.py                         @DataDog/ml-observability
tests/contrib/langchain/utils.py                                        @DataDog/ml-observability
tests/snapshots/tests.contrib.langchain.test_langchain.test_ai21_llm_sync.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_cohere_llm_sync.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_fake_embedding_document.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_fake_embedding_query.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_chat_model_sync_call_langchain_openai.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_embedding_query.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_llm_async.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_llm_error.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_llm_sync.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_llm_sync_multiple_prompts.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_pinecone_vectorstore_similarity_search.json  @DataDog/apm-python
tests/contrib/langchain/cassettes/ai21_completion_sync.yaml             @DataDog/ml-observability
tests/contrib/langchain/cassettes/anthropic_chat_completion_sync.yaml   @DataDog/ml-observability
tests/contrib/langchain/cassettes/bedrock_amazon_chat_invoke.yaml       @DataDog/ml-observability
tests/contrib/langchain/cassettes/bedrock_amazon_invoke.yaml            @DataDog/ml-observability
tests/contrib/langchain/cassettes/cohere_completion_sync.yaml           @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_acall.yaml          @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_batch.yaml          @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_batch_311.yaml      @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_batch_async.yaml    @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_call.yaml           @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_call_complicated.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_nested.yaml         @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chain_schema_io.yaml      @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chat_streamed_response.txt  @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_chat_streamed_response_json_output_parser.txt  @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_openai_llm_streamed_response.txt  @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_with_tools_anthropic.yaml        @DataDog/ml-observability
tests/contrib/langchain/cassettes/lcel_with_tools_openai.yaml           @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_chat_completion_async_call.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_chat_completion_async_generate.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_chat_completion_sync_call.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_chat_completion_sync_generate.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_completion_async.yaml          @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_completion_error.yaml          @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_completion_non_ascii.yaml      @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_completion_sync.yaml           @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_completion_sync_multi_prompt.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_embedding_query.yaml           @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_paraphrase.yaml                @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_pinecone_similarity_search.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_pinecone_similarity_search_community.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_pinecone_vectorstore_retrieval_chain.yaml  @DataDog/ml-observability
tests/contrib/langchain/cassettes/openai_retrieval_embedding.yaml       @DataDog/ml-observability
tests/snapshots/tests.contrib.langchain.test_langchain.test_base_tool_invoke.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_base_tool_invoke_non_json_serializable_config.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_faiss_vectorstore_retrieval.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_chain_batch.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_chain_complicated.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_chain_nested.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_chain_non_dict_input.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_chain_simple.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_chain_simple_async.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_with_tools_anthropic.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_lcel_with_tools_openai.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_chat_model_async_generate.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_chat_model_sync_generate.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_openai_chat_model_vision_generate.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_streamed_chain.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_streamed_chat.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_streamed_json_output_parser.json  @DataDog/apm-python
tests/snapshots/tests.contrib.langchain.test_langchain.test_streamed_llm.json  @DataDog/apm-python

datadog-dd-trace-py-rkomorn · 2025-01-30T16:59:35Z

Datadog Report

Branch report: sabrenner/langchain-drop-langchain-chains
Commit report: 92938e5
Test service: dd-trace-py

✅ 0 Failed, 130 Passed, 1378 Skipped, 5m 9.57s Total duration (35m 0.04s time saved)

tests/contrib/langchain/test_langchain.py

pr-commenter · 2025-01-30T18:00:43Z

Benchmarks

Benchmark execution time: 2025-01-31 14:52:53

Comparing candidate commit 92938e5 in PR branch sabrenner/langchain-drop-langchain-chains with baseline commit 9eeae42 in branch 3.x-staging.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 394 metrics, 2 unstable metrics.

tests/contrib/langchain/conftest.py

tests/contrib/langchain/test_langchain.py

Yun-Kim

Approving to unblock 3.0 but a couple suggestions to add for a follow up PR 🙇

releasenotes/notes/langchain-drop-langchain-base-chains-support-48d8290af6eb86e3.yaml

tests/contrib/langchain/test_langchain.py

…t-48d8290af6eb86e3.yaml

…ains

sabrenner added 4 commits January 30, 2025 11:44

remove patching

bdb76ed

remove patching tests

ab4c915

rel note

a5c329a

re-add tests

556a38e

sabrenner marked this pull request as ready for review January 30, 2025 16:55

sabrenner requested review from a team as code owners January 30, 2025 16:55

sabrenner requested review from Kyle-Verhoog and christophe-papazian January 30, 2025 16:55

sabrenner commented Jan 30, 2025

View reviewed changes

tests/contrib/langchain/test_langchain.py Show resolved Hide resolved

remove outdated test

9bc2f2e

sabrenner added 3 commits January 30, 2025 13:34

remove mock metrics and logs tests

998aea2

change snapshots and ignore field

1e812cf

fmt

5ed4775

emmettbutler approved these changes Jan 30, 2025

View reviewed changes

sabrenner added 5 commits January 30, 2025 14:54

try skipping flaky test, version issues

dbf238b

fix

6250483

fix vision test

3d3f253

ignore more problematic meta tags in snapshots

96e80fe

skip test

3d7ef71

sabrenner commented Jan 30, 2025

View reviewed changes

tests/contrib/langchain/conftest.py Show resolved Hide resolved

sabrenner commented Jan 30, 2025

View reviewed changes

tests/contrib/langchain/test_langchain.py Show resolved Hide resolved

sabrenner enabled auto-merge (squash) January 30, 2025 22:21

Yun-Kim approved these changes Jan 31, 2025

View reviewed changes

releasenotes/notes/langchain-drop-langchain-base-chains-support-48d8290af6eb86e3.yaml Outdated Show resolved Hide resolved

tests/contrib/langchain/test_langchain.py Show resolved Hide resolved

tests/contrib/langchain/test_langchain.py Show resolved Hide resolved

Yun-Kim and others added 3 commits January 30, 2025 19:14

Update releasenotes/notes/langchain-drop-langchain-base-chains-suppor…

bc6b414

…t-48d8290af6eb86e3.yaml

Merge branch '3.x-staging' into sabrenner/langchain-drop-langchain-ch…

d201535

…ains

Merge branch '3.x-staging' into sabrenner/langchain-drop-langchain-ch…

92938e5

…ains

sabrenner merged commit 7fbf54a into 3.x-staging Jan 31, 2025
563 checks passed

sabrenner deleted the sabrenner/langchain-drop-langchain-chains branch January 31, 2025 14:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(langchain): drop support for tracing `langchain.chains` #12172

chore(langchain): drop support for tracing `langchain.chains` #12172

sabrenner commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn bot commented Jan 30, 2025 •

edited

Loading

pr-commenter bot commented Jan 30, 2025 •

edited

Loading

Yun-Kim left a comment

chore(langchain): drop support for tracing langchain.chains #12172

chore(langchain): drop support for tracing langchain.chains #12172

Conversation

sabrenner commented Jan 30, 2025 • edited Loading

Checklist

Reviewer Checklist

github-actions bot commented Jan 30, 2025 • edited Loading

datadog-dd-trace-py-rkomorn bot commented Jan 30, 2025 • edited Loading

Datadog Report

pr-commenter bot commented Jan 30, 2025 • edited Loading

Benchmarks

Yun-Kim left a comment

Choose a reason for hiding this comment

chore(langchain): drop support for tracing `langchain.chains` #12172

chore(langchain): drop support for tracing `langchain.chains` #12172

sabrenner commented Jan 30, 2025 •

edited

Loading

github-actions bot commented Jan 30, 2025 •

edited

Loading

datadog-dd-trace-py-rkomorn bot commented Jan 30, 2025 •

edited

Loading

pr-commenter bot commented Jan 30, 2025 •

edited

Loading