Introduce otlp/http support

[TomasLongo]

Description

Introduce http endpoint to support otlp/http for the otel trace source. So far, this draft PR contains:

• A single armeria server listening on a port
• A HTTP Service
  • listening under /opentelemetry.proto.collector.trace.v1.TraceService/Export
  • processing ExportTraceServiceRequest
  • supporting basic auth/tls
• (not yet complete)Testing of the http service
• updated otel-proto specs

This draft PR serves as starting for further discussions

Configuration of HTTP and gRPC Service

Currently, the source config is responsible for setting up a working gRPC service. Now, an HTTP Service has to be configured as well.

As far as this PR is concerned, we can assign the current config items into two groups:

• Config items for gRPC specific features (e.g. unframed_requests, proto_reflection_service)
• Config items relevant for both services (e.g. compression, authentication) and should/could

This leads to the questions how the structure of the config should look like in the future

Create distinct sections for every endpoint

port: 123  
thread_count: 123  
...  
http:  
  path: /path  compression: gzip  authentication:    http_basic:grpc:  
  compression: gzip  proto_reflection_service: true  authentication:    http_basic:  

Let the endpoint share as much of the current config as possible

e.g. features like compression, authentication

Do we want to keep unframed requests

My understanding is that unframed requests enable clients to send plain http requests to the gRPC endpoint and would be rendered deprecated by this PR. However, there might be more to this feature which I'm currently unaware of.

Usage of the plugin mechanism to handle configs

The Method HttpService.createAuthenticationProvider contains a comment on how the handling of the basic auth config could be made less complex.

The basic idea is to not treat simple configs as plugins. The overall developer experience could benefit from using the simple parsing mechanism that is already used for other configs (like RetryInfo).

Again, there could be reasons for using the plugin mechanism which I'm not aware of.

Issues Resolved

Resolves #4983

Related to #5259

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[KarstenSchnitter]

Thanks for providing this PR. This is intended to start a discussion about this change. @dlvenable we would like your opinion as well as of all other maintainers. There are still a couple of TODOs in the comment, that need to be addressed. I think, they mark points waiting for clarification.

Tomas, can you explain a little bit more about the upgrade of the otlp-proto classes. Did you need to remove the instrumentation library support because of that? Do you know, how this change behaves if the old format is ingested?

[TomasLongo]

Tomas, can you explain a little bit more about the upgrade of the otlp-proto classes. Did you need to remove the instrumentation library support because of that?

Yes: According to the deprecation message in 0.16.0 InstrumentationLibrary was removed in June 15, 2022 (pretty specific). The now used version (1.3.2-alpha) does indeed not carry it anymore.

Do you know, how this change behaves if the old format is ingested?

My Tests so far:

• message carrying scopeSpans along with instrumentationLibrarySpans
  -- instrumentationLibrarySpans are simply ignored. The sink will receive the data from the scope spans
• message carrying only instrumentationLibrarySpans
  -- Sink will receive nothing, since the current implementation looks for scopeSpans inside the resourceSpans and returns an empty list if it can find none

All the tests have been made against the new http endpoint (using json as input). I have yet to test behaviour of the gRPC endpoint (I don't expect a different behaviour, though, since both endpoints use the same decoding functionality)

[UPDATE ON TESTING 2025/01/17]
Testing the gRPC endpoint brought no surprises. Clients with outdated protobuf specifications will still be able to use data-prepper as long as their messages come packed with ScopeSpans

The spec of 0.16.0 carried details on how senders and receivers should behave during the grace period:

During the grace period the following rules SHOULD be followed:

For Binary Protobufs

Binary Protobuf senders SHOULD NOT set instrumentation_library_spans. Instead
scope_spans SHOULD be set.

Binary Protobuf receivers SHOULD check if instrumentation_library_spans is set
and scope_spans is not set then the value in instrumentation_library_spans
SHOULD be used instead by converting InstrumentationLibrarySpans into ScopeSpans.
If scope_spans is set then instrumentation_library_spans SHOULD be ignored.

For JSON

JSON senders that set instrumentation_library_spans field MAY also set
scope_spans to carry the same spans, essentially double-publishing the same data.
Such double-publishing MAY be controlled by a user-settable option.
If double-publishing is not used then the senders SHOULD set scope_spans and
SHOULD NOT set instrumentation_library_spans.

JSON receivers SHOULD check if instrumentation_library_spans is set and
scope_spans is not set then the value in instrumentation_library_spans
SHOULD be used instead by converting InstrumentationLibrarySpans into ScopeSpans.
If scope_spans is set then instrumentation_library_spans field SHOULD be ignored.

According to that comment, the data-prepper behaves correctly now, and did adhere to the specfication in the past (by using either ScopeSpans or InstrumentationLibrarySpans).

IMO, if Data-Prepper clients followed the recommendations, there should be no friction upgrading the proto classes.

[TomasLongo]

I discovered a comment that could be important for backward compatibility in the 0.16.0 deprecation notice:

InstrumentationLibrarySpans is wire-compatible with ScopeSpans for binary Protobuf format

I'm not that fluent in protbuf speech yet to really grasp the implication of that statement. Does wire-compatible mean, that the binary representation of InstrumentationLibrarySpans and ScopeSpans does look the same and that receivers can therefore simply decode incoming InstrumentationLibrarySpans into ScopeSpans in their code without any further effort?

[KarstenSchnitter]

@TomasLongo I will check the wire compatibility. Can you have a look at the OTelTraceSource_*Test test cases. For me the first test-case in ./gradlew ":data-prepper-plugins:otel-trace-source:test" fails causing all subsequent tests to fail as well:

./gradlew ":data-prepper-plugins:otel-trace-source:test"

> Task :data-prepper-plugins:otel-trace-source:test

OTelTraceSourceTest > testOptionalHttpAuthServiceInPlaceWithUnauthenticatedDisabled() FAILED
    java.lang.NullPointerException: Cannot invoke "Object.toString()" because the return value of "java.util.Map.get(Object)" is null

OTelTraceSourceTest > testHealthCheckUnauthAllowed() FAILED
    java.lang.AssertionError: Http Status
    Expected: <200 OK>
         but: was <401 Unauthorized>

OTelTraceSourceTest > testServerConnectionsMetric() FAILED
    java.lang.RuntimeException: java.util.concurrent.ExecutionException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

        Caused by:
        java.util.concurrent.ExecutionException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

            Caused by:
            io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

OTelTraceSourceTest > testOptionalHttpAuthServiceInPlace() FAILED
    java.lang.NullPointerException: Cannot invoke "Object.toString()" because the return value of "java.util.Map.get(Object)" is null

OTelTraceSourceTest > testHttpFullJsonWithCustomPathAndAuthHeader_with_successful_response() FAILED
    java.lang.RuntimeException: java.util.concurrent.ExecutionException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

        Caused by:
        java.util.concurrent.ExecutionException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

            Caused by:
            io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

[...]

This is also causing the failing tests on the build system.

[KarstenSchnitter]

Check this line. This causes an NPE in testing, when pluginsSettings is an empty map. Run test case OTelTraceSourceTest.testOptionalHttpAuthServiceInPlaceWithUnauthenticatedDisabled.

[TomasLongo]

This is one of the two tests (the other beeing testOptionalHttpAuthServiOceInPlace) that causes NPEs. I disabled them for now! However, we should discuss if they are still needed.

The reason is, that the test assumes an invalid auth-config, which data-prepper refuses to ingest in production. The config in the test looks as follows:

otel_trace_source:
authentication:
test:
The tests are nonetheless green in main, because the critical code, the PluginFactory, is completely mocked away in the test. This branch produces NPEs, because the new HTTPService does not use the Pluginfactory to instantiate the AuthProvider.

IMO, this tests could be removed.

[KarstenSchnitter]

@dlvenable what is your take on that?

[dlvenable]

@TomasLongo , We use plugins for authentication to allow for custom authentication which may not always be available in the open source project. We should continue that pattern here as well. Also, we should avoid having passwords and usernames stored in the code.

[TomasLongo]

ok. done. The auth provider us now created via the plugin factory.

I'm not sure what you mean by "passwords and usernames stored in code", though. I could not find any credentials stored in the code.

[kkondaka]

@TomasLongo Some tests (like ':e2e-test:trace:compileIntegrationTestJava') are failing. Please check

[kkondaka]

@TomasLongo @KarstenSchnitter @dlvenable should we split this in to 2 PRs? One for just updated OTEL spec version? And another for OTLP_HTTP support?

[TomasLongo]

@kkondaka @KarstenSchnitter @dlvenable

As suggested, I've created a new PR #5434, containing only the updated otel proto buf spec. I've rebased this branch in order to simply create a patch of the changes for the new spec. You'll need to refetch the branch if you have checked it out before.

I'll cleanup this PR here and incorporate your suggestions.

However, we still have to address the configuration of the new http service. Any thoughts on that topic? Or should I come up with with a first draft for this?

[TomasLongo]

I've cleaned up the PR as far as I could, without needing further input from maintainer side. The remaining todos cover the following topics:

• how to design the otel-trace-source-config now that the http service entered the game
• do we still need unframed requests?

[TomasLongo]

Here's a first draft, of how the config might look like. The assumption is, that both endpoints serve the same purpose, so they should share as much config as possible. That's why things like compression, health_check_service, etc are shared.

source:
    otel_trace_source:
        retry_info:
        compression: 'none' | 'gzip'
        ssl: boolean
        request_timeout: int
        authentication:
        health_check_service: boolean
        http:
          path: string
        grpc:
          proto_reflection_service: boolean

the other option could be complete separation. e.g. every config property is duplicated per service, so that they can be configured independently.