build: add metadata descriptor to enable kafka connect use the plugin discovery mechanism #285
+301
−83
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eliax1996
force-pushed
the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
4 times, most recently
from
e7f3371 to
dd3ea20
Compare
eliax1996
commented
Sep 6, 2024
| @@ -186,12 +186,14 @@ tasks.distZip { dependsOn(":commons:jar") } | |||
| distributions { | |||
| main { | |||
| contents { | |||
| from("jar") | |||
| from(configurations.runtimeClasspath.get().map { if (it.isDirectory) it else zipTree(it) }) | |||
| from(tasks.jar) | |||
There was a problem hiding this comment.
wasn't adding the right folder to the installDist step
eliax1996
force-pushed
the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
from
dd3ea20 to
de7d201
Compare
gharris1727
previously approved these changes
Sep 6, 2024
eliax1996
force-pushed
the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
from
de7d201 to
6a21e5d
Compare
eliax1996
changed the title
eliax1996
changed the title
eliax1996
force-pushed
the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
5 times, most recently
from
e6369f7 to
f5d3f21
Compare
eliax1996
commented
Sep 10, 2024
eliax1996
force-pushed
the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
3 times, most recently
from
63c6605 to
09e9e4d
Compare
… discovery mechanism generating jars instead of class files in the `distInstall` (just keeping the runtime classpath and not extracting it with the zip plugin).
eliax1996
force-pushed
the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
from
09e9e4d to
839ba33
Compare
jjaakola-aiven
approved these changes
Sep 11, 2024
jjaakola-aiven
deleted the
eliax1996/add-META-INF-services-to-enable-service_load-loading
branch
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pr add the descriptors to enable kafka connect loading the connectors just by scanning the plugin jars.
The specific require creating a file called
META-INF/services/org.apache.kafka.connect.sink.SinkConnector(note well that this is the interface, not the implementation. I've done this error before).The content of the file should be the fully qualified classname of the file implementing the interface.
To check that the modification its accepted you can use a cli tool, e.g.
/opt/kafka-3.6/bin/connect-plugin-path.sh list --plugin-path /opt/myFancyFolder/kafka-connect-bigquery/.The output will be like:
In this example the
hasManifestits true just for thekafka-connect-bigquery/kcbq-connector-2.6.1-SNAPSHOT.jarmeaning that the jar containing the implementation underMETA-INF/services/org.apache.kafka.connect.sink.SinkConnectorhas 1 row with specifiedcom.wepay.kafka.connect.bigquery.BigQuerySinkConnector.While the other two classes under
/opt/myFancyFolder/connect-plugins/kafka-connect-bigquery/debezium-core-0.6.2.jarhave two transformations that can be scanned with a runtime scanner (basically checking with the reflection for each class if its implementing the method) but do not have a valid descriptor for them.NB with the descriptor the scan time its several orders of time faster for non trivial classpath (classpath with multiple big jars ~= 2/300 MB)