Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug fixes] Lambda - duplicate lambda spans + appsignals from unsampled spans #1000

Merged
merged 5 commits into from
Jan 15, 2025

Conversation

srprash
Copy link
Contributor

@srprash srprash commented Jan 14, 2025

Issue 1 - Duplicate lambda root spans

The instrumentation produces 2 lambda root spans upon each invocation of the function. This is a known issue in OTel: open-telemetry/opentelemetry-java-instrumentation#7808

Fix

  • Otel has fixed it in this PR but the change wasn't ported to the 1.x versions. So I took the diff from the upstream commit and made it into a patch that we apply when building the ADOT Java Lambda Layer for v1.33.x.

Issue 2 - Unsampled spans do not produce Application Signals metrics

On lambda environment, we export 100% of the spans to X-Ray to ensure we are able to provide 100% Application Signals metrics. However, currently only the sampled spans show up on the "Services" page and the unsampled spans do not.

Fix

  • Upon comparing the sampled vs unsampled spans, I noticed that the unsampled spans are missing the attributes like aws.local.service and aws.local.operation which are required to generate Application Signals metrics.
  • The fix is to wrap the OtlpUdpSpanExporter instance for unsampled spans with the AwsMetricAttributesSpanExporter so that the exported spans have the desired attributes.

Testing

  • After creating a layer with the fix, I set the OTEL_TRACES_SAMPLER to always_off. Then I invoked the function once.
  • The metrics appeared on the Application Signals console.
  • See the following screenshots
image image

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@srprash srprash requested a review from a team as a code owner January 14, 2025 05:52
mxiamxia
mxiamxia previously approved these changes Jan 14, 2025
@srprash srprash merged commit c8bce79 into release/v1.33.x Jan 15, 2025
4 checks passed
@srprash srprash deleted the lambda_fixes_1.33.x branch January 15, 2025 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants