diff --git a/README.md b/README.md index 744d2db..89d8d2f 100644 --- a/README.md +++ b/README.md @@ -136,17 +136,6 @@ of the column can be `Int, Float, Decimal, Timestamp`. This considers SAP DB Tim * `{topic}.partition.count` - This setting can be used to specify the no. of topic partitions that the Source connector can use to publish the data. Should be an `Integer`. Default value is `1`. -#### Sample Configurations -- Source and Sink Properties - - Sample Connectors: [connect-hana-source.properties](config/connect-hana-source.properties),[connect-hana-sink.properties](config/connect-hana-sink.properties) - - PERSONS1 batch-mode Connetors: [connect-hana-source-1.properties](config/connect-hana-source-1.properties),[connect-hana-sink-1.properties](config/connect-hana-sink-1.properties) - - PERSONS2 incrementing-mode Connectors: [connect-hana-source-2.properties](config/connect-hana-source-2.properties),[connect-hana-sink-2.properties](config/connect-hana-sink-2.properties) -- Connector Properties - - Standalone Connector with JSON: [connect-standalone.properties](config/connect-standalone.properties) - - Standalone Connector with Avro using Confluent Schemas Registry: [connect-avro-confluent.properties](config/connect-avro-confluent.properties) - - Standalone Connector with Avro using Apicurio Schemas Registry: [connect-avro-apicurio.properties](config/connect-avro-apicurio.properties) - - ## Examples Folder [`examples`](examples) includes some example scenarios. In addtion, the `unit tests` provide examples on every possible mode in which the connector can be configured. @@ -157,7 +146,7 @@ We welcome comments, questions, and bug reports. Please [create an issue](https: ## Contributing -Contributions are accepted by sending Pull Requests to this repo. +Contributions are accepted by sending Pull Requests to this repo. Please do not forget to sign the [Contribution License Agreement](https://cla-assistant.io/SAP/kafka-connect-sap). ## Todos diff --git a/config/connect-hana-sink-1.properties b/config/connect-hana-sink-1.properties index b65e1ab..68abc7d 100644 --- a/config/connect-hana-sink-1.properties +++ b/config/connect-hana-sink-1.properties @@ -5,7 +5,7 @@ name=test_topic_1_sink connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector tasks.max=1 topics=test_topic_1 -connection.url=jdbc:sap:/// +connection.url=jdbc:sap:/// connection.user= connection.password= auto.create=true diff --git a/config/connect-hana-sink-2.properties b/config/connect-hana-sink-2.properties index 276e495..6ceecf2 100644 --- a/config/connect-hana-sink-2.properties +++ b/config/connect-hana-sink-2.properties @@ -5,7 +5,7 @@ name=test_topic_2_sink connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector tasks.max=1 topics=test_topic_2 -connection.url=jdbc:sap:/// +connection.url=jdbc:sap:/// connection.user= connection.password= auto.create=true diff --git a/config/connect-hana-sink.properties b/config/connect-hana-sink.properties index 701d427..1f3f55e 100644 --- a/config/connect-hana-sink.properties +++ b/config/connect-hana-sink.properties @@ -3,7 +3,7 @@ connector.class=com.sap.kafka.connect.sink.hana.HANASinkConnector tasks.max=2 topics=test_topic1,test_topic2 -connection.url=jdbc:sap:/// +connection.url=jdbc:sap:/// connection.user= connection.password= batch.size=3000 diff --git a/config/connect-hana-source-1.properties b/config/connect-hana-source-1.properties index 0f87c23..62649cc 100644 --- a/config/connect-hana-source-1.properties +++ b/config/connect-hana-source-1.properties @@ -6,7 +6,7 @@ name=test-topic-1-source connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector tasks.max=1 topics=test_topic_1 -connection.url=jdbc:sap:/// +connection.url=jdbc:sap:/// connection.user= connection.password= test_topic_1.table.name=."PERSONS1" diff --git a/config/connect-hana-source-2.properties b/config/connect-hana-source-2.properties index 4469d32..fd008b3 100644 --- a/config/connect-hana-source-2.properties +++ b/config/connect-hana-source-2.properties @@ -6,7 +6,7 @@ name=test-topic-2-source connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector tasks.max=1 topics=test_topic_2 -connection.url=jdbc:sap:/// +connection.url=jdbc:sap:/// connection.user= connection.password= mode=incrementing diff --git a/config/connect-hana-source.properties b/config/connect-hana-source.properties index 784db30..f054c1d 100644 --- a/config/connect-hana-source.properties +++ b/config/connect-hana-source.properties @@ -3,7 +3,7 @@ connector.class=com.sap.kafka.connect.source.hana.HANASourceConnector tasks.max=1 topics=test_topic1,test_topic2 -connection.url=jdbc:sap:/// +connection.url=jdbc:sap:/// connection.user= connection.password= mode=incrementing diff --git a/examples/README.md b/examples/README.md index c4010bc..17f6bcd 100644 --- a/examples/README.md +++ b/examples/README.md @@ -5,3 +5,6 @@ * [persons3](persons3) : standalone Source and Sink HANA-Connectors using Apicurio Schema Registry (JSON) * [persons4](persons4) : standalone Source and Sink HANA-Connectors using Apicurio Schema Registry (Avro) * [persons6](persons6) : standalone Source and Sink HANA-Connectors using Confluent Schema Registry (Avro) +* [persons1ds](persons1ds) : distributed version of persons1 using Docker with Strimzi Kafka image +* [persons1db](persons1db) : distributed version of persons1 using Docker with Debezium Kafka-Connect image +* [persons4ds](persons4ds) : distributed version of persons4 using Docker with Strimzi Kafka image diff --git a/examples/persons1/Makefile b/examples/persons1/Makefile new file mode 100644 index 0000000..0b29172 --- /dev/null +++ b/examples/persons1/Makefile @@ -0,0 +1,13 @@ +# +# +# make get_libs - build the go binary +# + +DOCKER_TAG ?= latest + +.PHONY: get_libs +get_libs: + @echo "Getting jar files into target ..." + @mkdir -p target + @cp ../../target/kafka-connect-hana-*.jar target + @cat driver-jars.txt | xargs -I '{}' mvn -q dependency:get -Dartifact='{}' -Dtransitive=false -Ddest=target diff --git a/examples/persons1/README.md b/examples/persons1/README.md index b0ddb08..b28fbcb 100644 --- a/examples/persons1/README.md +++ b/examples/persons1/README.md @@ -31,30 +31,25 @@ For more information regarding how to start Kafka, refer to https://kafka.apache We install the jar files into a dedicated directory within the plugins directory `plugins` that we create at `$KAFKA_HOME`. -First, we create a plugins directory `$KAFKA_HOME/plugins` if not yet created and create directory `kafka-connect-hana` within this directory. Assuming this project has been built (see Building), copy the connector's jar file into this directory. - +First, we create a plugins directory `$KAFKA_HOME/plugins` if not yet created and create directory `kafka-connect-hana` within this directory. ``` $ mkdir -p $KAFKA_HOME/plugins/kafka-connect-hana -$ cp $KAFKA_CONNECT_SAP/target/kafka-connect-hana-1.0-SNAPSHOT.jar $KAFKA_HOME/plugins/kafka-connect-hana $ ``` -Download the the HANA jdbc driver and put it into the target `$KAFKA_HOME/plugins/kafka-connect-hana` directory. +Assuming this project has been built (see Building), run `make get_libs` to place the required jar files including the HANA jdbc driver into directory 'target'. ``` -$ wget https://repo1.maven.org/maven2/com/sap/cloud/db/jdbc/ngdbc/2.5.49/ngdbc-2.5.49.jar ---2020-07-24 20:14:06-- https://repo1.maven.org/maven2/com/sap/cloud/db/jdbc/ngdbc/2.5.49/ngdbc-2.5.49.jar -Resolving repo1.maven.org (repo1.maven.org)... 151.101.112.209 -Connecting to repo1.maven.org (repo1.maven.org)|151.101.112.209|:443... connected. -HTTP request sent, awaiting response... 200 OK -Length: 1219123 (1.2M) [application/java-archive] -Saving to: 'ngdbc-2.5.49.jar' - -ngdbc-2.5.49.jar 100%[===========================================>] 1.16M 5.23MB/s in 0.2s - -2020-07-24 20:14:06 (5.23 MB/s) - 'ngdbc-2.5.49.jar' saved [1219123/1219123] +$ make get_libs +Getting jar files into target ... +$ ls target +kafka-connect-hana-1.0-SNAPSHOT.jar ngdbc-2.5.49.jar +$ +``` +Copy those jar files into `$KAFKA_HOME/plugins/kafka-connect-hana` directory. -$ mv ngdbc-2.5.49.jar $KAFKA_HOME/plugins/kafka-connect-hana +``` +$ cp target/*.jar $KAFKA_HOME/plugins/kafka-connect-hana $ ``` diff --git a/examples/persons1/driver-jars.txt b/examples/persons1/driver-jars.txt new file mode 100644 index 0000000..1040f1e --- /dev/null +++ b/examples/persons1/driver-jars.txt @@ -0,0 +1 @@ +com.sap.cloud.db.jdbc:ngdbc:2.5.49 diff --git a/examples/persons1db/Dockerfile b/examples/persons1db/Dockerfile new file mode 100644 index 0000000..df0ed79 --- /dev/null +++ b/examples/persons1db/Dockerfile @@ -0,0 +1,7 @@ +FROM debezium/connect:1.2 +USER root:root + +RUN mkdir -p /kafka/connect/kafka-connector-hana +COPY ./target/ /kafka/connect/kafka-connector-hana/ + +USER 1001 diff --git a/examples/persons1db/Makefile b/examples/persons1db/Makefile new file mode 100644 index 0000000..fb396b0 --- /dev/null +++ b/examples/persons1db/Makefile @@ -0,0 +1,19 @@ +# +# +# make get_libs - build the go binary +# make docker_build - build the docker image +# + +DOCKER_TAG ?= latest + +.PHONY: get_libs +get_libs: + @echo "Getting jar files into target ..." + @mkdir -p target + @cp ../../target/kafka-connect-hana-*.jar target + @cat driver-jars.txt | xargs -I '{}' mvn -q dependency:get -Dartifact='{}' -Dtransitive=false -Ddest=target + +.PHONY: docker_build +docker_build: + @echo "Building docker image ..." + docker build . -t debezium-connector-hana-min diff --git a/examples/persons1db/README.md b/examples/persons1db/README.md new file mode 100644 index 0000000..b0fc4b8 --- /dev/null +++ b/examples/persons1db/README.md @@ -0,0 +1,220 @@ +### Example persons1db: kafka-hana-connect using debezium kafka images with docker + +This example is a distributed version of example persons1 using debezium kafka-connect docker images. + +#### Prerequisites + +- This project is built (or its jar file is available) +- Access to HANA +- Docker + +#### Running + +This description assumes Docker and Docker-Compose are available on local machine. + +##### Step 1: Start Zookeeper and Kafka + +First, we start both Zookeeper and Kafka using debezium zookeeper and kafka images. If you are not familiar with debezium, you can find more information at Debezim tutorial https://debezium.io/documentation/reference/1.2/tutorial.html + +Start Zookeeper using the following `docker run` command. + +``` +$ docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.2 +Starting up in standalone mode +ZooKeeper JMX enabled by default +Using config: /zookeeper/conf/zoo.cfg +2020-09-09 22:42:33,018 - INFO [main:QuorumPeerConfig@135] - Reading configuration from: /zookeeper/conf/zoo.cfg +2020-09-09 22:42:33,031 - INFO [main:QuorumPeerConfig@387] - clientPortAddress is 0.0.0.0:2181 +... +``` + +Start Kafka using the following `docker run` command. + +``` +$ docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.2 +WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations. +Using ZOOKEEPER_CONNECT=172.17.0.2:2181 +Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.3:9092 +2020-09-09 22:43:05,396 - INFO [main:Log4jControllerRegistration$@31] - Registered kafka:type=kafka.Log4jController MBean +2020-09-09 22:43:05,934 - INFO [main:X509Util@79] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation +... +``` + +Before we start Kafka-Connect, we must prepare the Docker image that contains kafka-connect-hana. + +##### Steps 2: Build Docker image for kafka-connect-hana + +First, run `make get_libs` to place the required jar files into directory `target`. + +``` +$ make get_libs +Getting jar files into target ... +$ +$ ls target +guava-20.0.jar ngdbc-2.5.49.jar +kafka-connect-hana-1.0-SNAPSHOT.jar +$ +``` + +Next, run `make build_docker` to build the Docker image using debezium's Kafka Connect image and add the jar files to its plugins directory. + + +``` +$ make docker_build +Building docker image ... +docker build . -t debezium-connector-hana-min +Sending build context to Docker daemon 3.868MB +Step 1/5 : FROM debezium/connect:1.2 + ---> 66f074fce2f0 +Step 2/5 : USER root:root + ---> Using cache + ---> 64a9079cfa93 +Step 3/5 : RUN mkdir -p /kafka/connect/kafka-connector-hana + ---> Using cache + ---> 174c3e1fb6db +Step 4/5 : COPY ./target/ /kafka/connect/kafka-connector-hana/ + ---> Using cache + ---> 8f300532bf25 +Step 5/5 : USER 1001 + ---> Using cache + ---> b38cc3546555 +Successfully built b38cc3546555 +Successfully tagged debezium-connector-hana-min:latest +$ +``` + +##### Step 3: Prepare the connector configuration files (Follow Step 2 of [persons1ds example](../persons1ds). + +##### Step 4: Prepare the source table (Follow Step 4 of example persons1) + +##### Step 5: Starting Kafka-Connect + +Start Kafka-Connect using the following `docker run` command. + +``` +$ docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka debezium-connector-hana-min:latest +Plugins are loaded from /kafka/connect +Using the following environment variables: + GROUP_ID=1 + CONFIG_STORAGE_TOPIC=my_connect_configs + OFFSET_STORAGE_TOPIC=my_connect_offsets + STATUS_STORAGE_TOPIC=my_connect_statuses + BOOTSTRAP_SERVERS=172.17.0.3:9092 + REST_HOST_NAME=172.17.0.4 +... +``` + +After starting those Docker containers, we can verify whether Kafka-Connect is running using curl. + +``` +$ curl -i http://localhost:8083/ +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:44:49 GMT +Content-Type: application/json +Content-Length: 91 +Server: Jetty(9.4.24.v20191120) + +{"version":"2.5.0","commit":"66563e712b0b9f84","kafka_cluster_id":"1NEvm9a4TW2t-f5Jkk4peg"} +$ +$ curl -i http://localhost:8083/connectors +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:45:35 GMT +Content-Type: application/json +Content-Length: 2 +Server: Jetty(9.4.24.v20191120) + +[] +$ +``` + +The above result shows that Kafka Connect using Kafka 2.5.0 is running and there is no connector deployed. + +We prepare for the connector json files using the json files `connect-hana-source-1.json` and `connect-hana-sink-1.json` which are the json representation of the configuration files created for example persons1. + +Finally, we deploy the connectors by posting the connector configuration json files to the Kafka Connect's API. Assuming, these json files are already prepared in step 3, use curl to post these files. + +``` +$ curl -i -X POST -H 'content-type:application/json' -d @connect-hana-source-1.json http://localhost:8083/connectors +HTTP/1.1 201 Created +Date: Wed, 09 Sep 2020 22:46:30 GMT +Location: http://localhost:8083/connectors/test-topic-1-source +Content-Type: application/json +Content-Length: 399 +Server: Jetty(9.4.24.v20191120) + +{"name":"test-topic-1-source","config":{"connector.class":"com.sap.kafka.connect.source.hana.HANASourceConnector","tasks.max":"1","topics":"test_topic_1","connection.url":"jdbc:sap://... +$ +$ curl -i -X POST -H 'content-type:application/json' -d @connect-hana-sink-1.json http://localhost:8083/connectors +HTTP/1.1 201 Created +Date: Wed, 09 Sep 2020 22:46:39 GMT +Location: http://localhost:8083/connectors/test-topic-1-sink +Content-Type: application/json +Content-Length: 414 +Server: Jetty(9.4.24.v20191120) + +{"name":"test-topic-1-sink","config":{"connector.class":"com.sap.kafka.connect.sink.hana.HANASinkConnector","tasks.max":"1","topics":"test_topic_1","connection.url":"jdbc:sap://... +$ +$ curl -i http://localhost:8083/connectors +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:47:35 GMT +Content-Type: application/json +Content-Length: 43 +Server: Jetty(9.4.24.v20191120) + +["test-topic-1-source","test-topic-1-sink"] +$ +``` + +The above result shows that the connectors are deployed. + +##### Step 6: Verifying the result + +You can use debezium's watch-topic utility to look at the topic. + +``` +$ docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.2 watch-topic -a -k test_topic_1 +WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations. +Using ZOOKEEPER_CONNECT=172.17.0.2:2181 +Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.5:9092 +Using KAFKA_BROKER=172.17.0.3:9092 +Contents of topic test_topic_1: +null {"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"PERSONID"},{"type":"string","optional":true,"field":"LASTNAME"},{"type":"string","optional":true,"field":"FIRSTNAME"}],"optional":false,"name":"d025803persons1"},"payload":{"PERSONID":1,"LASTNAME":"simpson","FIRSTNAME":"homer"}} +null {"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"PERSONID"},{"type":"string","optional":true,"field":"LASTNAME"},{"type":"string","optional":true,"field":"FIRSTNAME"}],"optional":false,"name":"d025803persons1"},"payload":{"PERSONID":2,"LASTNAME":"simpson","FIRSTNAME":"merge"}} +... +``` + +You can also start another kafka container an interactive bash shell to inspect the topic using kafka-consule-consumer.sh. + +First, start the interactive bash shell with debezium kafka image + +``` +$ docker run -it --rm --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.2 /bin/bash +WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations. +Using ZOOKEEPER_CONNECT=172.17.0.2:2181 +Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.5:9092 +[kafka@2f92f972f2b6 ~]$ +``` + +In the interactive shell, use kafka-console-consumer.sh to connect kafka:9092 to inspect the topic. + +``` +[kafka@2f92f972f2b6 ~]$ bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test_topic_1 --from-beginning +{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"PERSONID"},{"type":"string","optional":true,"field":"LASTNAME"},{"type":"string","optional":true,"field":"FIRSTNAME"}],"optional":false,"name":"d025803persons1"},"payload":{"PERSONID":1,"LASTNAME":"simpson","FIRSTNAME":"homer"}} +{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"PERSONID"},{"type":"string","optional":true,"field":"LASTNAME"},{"type":"string","optional":true,"field":"FIRSTNAME"}],"optional":false,"name":"d025803persons1"},"payload":{"PERSONID":2,"LASTNAME":"simpson","FIRSTNAME":"merge"}} +... +``` + +It is noted that this scenario builds the Docker image without schema registry usage and runs Kafka Connect in the distributed mode. Additional connectors can be deployed to this Kafka Connect instance which use the same distributed-connect.properties configuration. + +##### Step 7: Cleaning up + +Use docker stop to terminate the containers. + +``` +$ docker stop watcher connect kafka zookeeper +watcher +connect +kafka +zookeeper +$ +``` diff --git a/examples/persons1db/driver-jars.txt b/examples/persons1db/driver-jars.txt new file mode 100644 index 0000000..e1ec8d4 --- /dev/null +++ b/examples/persons1db/driver-jars.txt @@ -0,0 +1,2 @@ +com.sap.cloud.db.jdbc:ngdbc:2.5.49 +com.google.guava:guava:20.0 diff --git a/examples/persons1ds/Dockerfile b/examples/persons1ds/Dockerfile new file mode 100644 index 0000000..6367a1f --- /dev/null +++ b/examples/persons1ds/Dockerfile @@ -0,0 +1,7 @@ +FROM strimzi/kafka:0.19.0-kafka-2.4.1 +USER root:root + +RUN mkdir -p /opt/kafka/plugins/kafka-connect-hana +COPY ./target/ /opt/kafka/plugins/kafka-connect-hana/ + +USER 1001 diff --git a/examples/persons1ds/Makefile b/examples/persons1ds/Makefile new file mode 100644 index 0000000..3d8ccd6 --- /dev/null +++ b/examples/persons1ds/Makefile @@ -0,0 +1,19 @@ +# +# +# make get_libs - build the go binary +# make docker_build - build the docker image +# + +DOCKER_TAG ?= latest + +.PHONY: get_libs +get_libs: + @echo "Getting jar files into target ..." + @mkdir -p target + @cp ../../target/kafka-connect-hana-*.jar target + @cat driver-jars.txt | xargs -I '{}' mvn -q dependency:get -Dartifact='{}' -Dtransitive=false -Ddest=target + +.PHONY: docker_build +docker_build: + @echo "Building docker image ..." + docker build . -t strimzi-connector-hana-min diff --git a/examples/persons1ds/README.md b/examples/persons1ds/README.md new file mode 100644 index 0000000..e0493ce --- /dev/null +++ b/examples/persons1ds/README.md @@ -0,0 +1,158 @@ +### Example persons1ds: kafka-hana-connect using strimzi kafka images with docker-compose + +This example is a distributed version of example persons1 using strimizi kafka docker images. + +#### Prerequisites + +- This project is built (or its jar file is available) +- Access to HANA +- Docker, Docker-Compose + +#### Running + +This description assumes Docker and Docker-Compose are available on local machine. + +##### Steps 1: Build Docker image for kafka-connect-hana + +First, run `make get_libs` to place the required jar files into directory `target`. + +``` +$ make get_libs +Getting jar files into target ... +$ +$ ls target +kafka-connect-hana-1.0-SNAPSHOT.jar ngdbc-2.5.49.jar +$ +``` + +Next, run `make build_docker` to build the Docker image using strimzi's Kafka image and add the jar files to its plugins directory. + +``` +$ make docker_build +Building docker image ... +docker build . -t strimzi-connector-hana-min +Sending build context to Docker daemon 1.423MB +Step 1/5 : FROM strimzi/kafka:0.19.0-kafka-2.4.1 + ---> a0be3ed644fc +Step 2/5 : USER root:root + ---> Using cache + ---> e651c111cfd0 +Step 3/5 : RUN mkdir -p /opt/kafka/plugins/kafka-connect-hana + ---> Using cache + ---> b5e5ec331af9 +Step 4/5 : COPY ./target/ /opt/kafka/plugins/kafka-connect-hana/ + ---> Using cache + ---> d04b23041483 +Step 5/5 : USER 1001 + ---> Using cache + ---> 44fb0e9e5347 +Successfully built 44fb0e9e5347 +Successfully tagged strimzi-connector-hana-min:latest +$ +``` + +##### Step 2: Prepare the connector configuration files + +We use connect-distributed.properties stored in directory custom-config for configuring Kafka-Connect. For configuring connectors, prepare the json version of the connector configuration files connect-hana-source-1.json and connect-hana-sink-1.json. + +##### Step 3: Prepare the source table (Follow Step 4 of example persons1) + +##### Step 4: Starting Zookeeper, Kafka, Kafka-Coonnect + +The docker-compose.yaml file defines zookeeper, kafka, and connect services. It is noted that Kafka broker uses its advertised host set to `host.docker.internal:9092` assumeing this host name is resolvable from the containers and at the host. This allows Kafka broker to be accessed from the container of Kafka-Connect and from the host for inspection. + +Run `docker-compose up` to start the containers. + +``` +$ docker-compose up +Creating network "persons1ds_default" with the default driver +Creating persons1ds_zookeeper_1 ... done +Creating persons1ds_kafka_1 ... done +Creating persons1ds_connect_1 ... done +Attaching to persons1ds_zookeeper_1, persons1ds_kafka_1, persons1ds_connect_1 +... +``` + +After starting the Docker containers using docker-compose, we can verify whether Kafka Connect is running using curl. + +``` +$ curl -i http://localhost:8083/ +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:00:11 GMT +Content-Type: application/json +Content-Length: 91 +Server: Jetty(9.4.20.v20190813) + +{"version":"2.4.1","commit":"c57222ae8cd7866b","kafka_cluster_id":"wdrDgSAFSbKpWGYm-q0PuQ"} +$ +$ curl -i http://localhost:8083/connectors +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:00:39 GMT +Content-Type: application/json +Content-Length: 2 +Server: Jetty(9.4.20.v20190813) + +[] +$ +``` + +The above result shows that Kafka Connect using Kafka 2.4.1 is running and there is no connector deployed. + +We prepare for the connector json files using the json files `connect-hana-source-1.json` and `connect-hana-sink-1.json` which are the json representation of the property files created for example persons1. + +Finally, we deploy the connectors by posting the connector configuration json files to the Kafka Connect's API. Assuming, these json files are already prepared in step 3, use curl to post these files. + +``` +$ curl -i -X POST -H 'content-type:application/json' -d @connect-hana-source-1.json http://localhost:8083/connectors +HTTP/1.1 201 Created +Date: Wed, 09 Sep 2020 22:10:34 GMT +Location: http://localhost:8083/connectors/test-topic-1-source +Content-Type: application/json +Content-Length: 399 +Server: Jetty(9.4.20.v20190813) + +{"name":"test-topic-1-source","config":{"connector.class":"com.sap.kafka.connect.source.hana.HANASourceConnector","tasks.max":"1","topics":"test_topic_1","connection.url":"jdbc:sap://... +$ +$ curl -i -X POST -H 'content-type:application/json' -d @connect-hana-sink-1.json http://localhost:8083/connectors +HTTP/1.1 201 Created +Date: Wed, 09 Sep 2020 22:11:22 GMT +Location: http://localhost:8083/connectors/test-topic-1-sink +Content-Type: application/json +Content-Length: 414 +Server: Jetty(9.4.20.v20190813) + +{"name":"test-topic-1-sink","config":{"connector.class":"com.sap.kafka.connect.sink.hana.HANASinkConnector","tasks.max":"1","topics":"test_topic_1","connection.url":"jdbc:sap://... +$ +$ curl -i http://localhost:8083/connectors +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:11:54 GMT +Content-Type: application/json +Content-Length: 43 +Server: Jetty(9.4.20.v20190813) + +["test-topic-1-source","test-topic-1-sink"] +``` + +The above result shows that the connectors are deployed. + + +##### Step 5: Verifying the result (Follow Step 6 of example persions1 and/or persons2) + +It is noted that this scenario builds the Docker image without schema registry usage and runs Kafka Connect in the distributed mode. Additional connectors can be deployed to this Kafka Connect instance which use the same distributed-connect.properties configuration. + + +##### Step 6: Shut down + +Use `docker-compose down` to shutdown the containers. + +``` +$ docker-compose down +Stopping persons1ds_connect_1 ... done +Stopping persons1ds_kafka_1 ... done +Stopping persons1ds_zookeeper_1 ... done +Removing persons1ds_connect_1 ... done +Removing persons1ds_kafka_1 ... done +Removing persons1ds_zookeeper_1 ... done +Removing network persons1ds_default +$ +``` diff --git a/examples/persons1ds/connect-hana-sink-1.json b/examples/persons1ds/connect-hana-sink-1.json new file mode 100644 index 0000000..41e9d2a --- /dev/null +++ b/examples/persons1ds/connect-hana-sink-1.json @@ -0,0 +1,14 @@ +{ + "name": "test-topic-1-sink", + "config": { + "connector.class": "com.sap.kafka.connect.source.hana.HANASourceConnector", + "tasks.max": "1", + "topics": "test_topic_1", + "connection.url": "jdbc:sap:///", + "connection.user": "", + "connection.password": "", + "auto.create": "true", + "test_topic_1.table.name": "\"\".\"PERSONS1_RES\"" + } +} + diff --git a/examples/persons1ds/connect-hana-source-1.json b/examples/persons1ds/connect-hana-source-1.json new file mode 100644 index 0000000..dc98ac1 --- /dev/null +++ b/examples/persons1ds/connect-hana-source-1.json @@ -0,0 +1,12 @@ +{ + "name": "test-topic-1-source", + "config": { + "connector.class": "com.sap.kafka.connect.source.hana.HANASourceConnector", + "tasks.max": "1", + "topics": "test_topic_1", + "connection.url": "jdbc:sap:///", + "connection.user": "", + "connection.password": "", + "test_topic_1.table.name": "\"\".\"PERSONS1\"" + } +} diff --git a/examples/persons1ds/custom-config/connect-distributed.properties b/examples/persons1ds/custom-config/connect-distributed.properties new file mode 100644 index 0000000..a7e724b --- /dev/null +++ b/examples/persons1ds/custom-config/connect-distributed.properties @@ -0,0 +1,86 @@ +## +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## + +# This file contains some of the configurations for the Kafka Connect distributed worker. This file is intended +# to be used with the examples, and some settings may differ from those used in a production system, especially +# the `bootstrap.servers` and those specifying replication factors. + +# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. +bootstrap.servers=kafka:9092 + +# unique name for the cluster, used in forming the Connect cluster group. Note that this must not conflict with consumer group IDs +group.id=connect-cluster + +# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will +# need to configure these based on the format they want their data in when loaded from or stored into Kafka +key.converter=org.apache.kafka.connect.json.JsonConverter +value.converter=org.apache.kafka.connect.json.JsonConverter +# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply +# it to +key.converter.schemas.enable=true +value.converter.schemas.enable=true + +# Topic to use for storing offsets. This topic should have many partitions and be replicated and compacted. +# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create +# the topic before starting Kafka Connect if a specific topic configuration is needed. +# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value. +# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able +# to run this example on a single-broker cluster and so here we instead set the replication factor to 1. +offset.storage.topic=connect-offsets +offset.storage.replication.factor=1 +#offset.storage.partitions=25 + +# Topic to use for storing connector and task configurations; note that this should be a single partition, highly replicated, +# and compacted topic. Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create +# the topic before starting Kafka Connect if a specific topic configuration is needed. +# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value. +# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able +# to run this example on a single-broker cluster and so here we instead set the replication factor to 1. +config.storage.topic=connect-configs +config.storage.replication.factor=1 + +# Topic to use for storing statuses. This topic can have multiple partitions and should be replicated and compacted. +# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create +# the topic before starting Kafka Connect if a specific topic configuration is needed. +# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value. +# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able +# to run this example on a single-broker cluster and so here we instead set the replication factor to 1. +status.storage.topic=connect-status +status.storage.replication.factor=1 +#status.storage.partitions=5 + +# Flush much faster than normal, which is useful for testing/debugging +offset.flush.interval.ms=10000 + +# These are provided to inform the user about the presence of the REST host and port configs +# Hostname & Port for the REST API to listen on. If this is set, it will bind to the interface used to listen to requests. +#rest.host.name= +rest.port=8083 + +# The Hostname & Port that will be given out to other workers to connect to i.e. URLs that are routable from other servers. +#rest.advertised.host.name= +#rest.advertised.port= + +# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins +# (connectors, converters, transformations). The list should consist of top level directories that include +# any combination of: +# a) directories immediately containing jars with plugins and their dependencies +# b) uber-jars with plugins and their dependencies +# c) directories immediately containing the package directory structure of classes of plugins and their dependencies +# Examples: +# plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors, +plugin.path=/opt/kafka/plugins diff --git a/examples/persons1ds/docker-compose.yaml b/examples/persons1ds/docker-compose.yaml new file mode 100644 index 0000000..36adf3c --- /dev/null +++ b/examples/persons1ds/docker-compose.yaml @@ -0,0 +1,48 @@ +version: '2' + +services: + + zookeeper: + image: strimzi/kafka:0.19.0-kafka-2.4.1 + command: [ + "sh", "-c", + "bin/zookeeper-server-start.sh config/zookeeper.properties" + ] + ports: + - "2181:2181" + environment: + LOG_DIR: /tmp/logs + + kafka: + image: strimzi/kafka:0.19.0-kafka-2.4.1 + command: [ + "sh", "-c", + "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}" + ] + depends_on: + - zookeeper + ports: + - "9092:9092" + environment: + LOG_DIR: "/tmp/logs" + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://host.docker.internal:9092 + KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 + KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 + + connect: + image: strimzi-connector-hana-min:latest + volumes: + - ./custom-config:/opt/kafka/custom-config + command: [ + "sh", "-c", + "bin/connect-distributed.sh custom-config/connect-distributed.properties" + ] + depends_on: + - kafka + ports: + - "8083:8083" + environment: + LOG_DIR: "/tmp/logs" + +volumes: + custom-config: \ No newline at end of file diff --git a/examples/persons1ds/driver-jars.txt b/examples/persons1ds/driver-jars.txt new file mode 100644 index 0000000..1040f1e --- /dev/null +++ b/examples/persons1ds/driver-jars.txt @@ -0,0 +1 @@ +com.sap.cloud.db.jdbc:ngdbc:2.5.49 diff --git a/examples/persons2/README.md b/examples/persons2/README.md index af739d7..6980b19 100644 --- a/examples/persons2/README.md +++ b/examples/persons2/README.md @@ -18,7 +18,7 @@ This description assumes Kafka 2.4.1 is installed on local machine and environme - start Kafka - install kafka-connect-hana -o##### Step 3: Prepare the connector configuration files +##### Step 3: Prepare the connector configuration files We use the provided configuration files and customize some properties. First, copy the source connector configuration file to the target `config` directory. diff --git a/examples/persons3/Makefile b/examples/persons3/Makefile new file mode 100644 index 0000000..0b29172 --- /dev/null +++ b/examples/persons3/Makefile @@ -0,0 +1,13 @@ +# +# +# make get_libs - build the go binary +# + +DOCKER_TAG ?= latest + +.PHONY: get_libs +get_libs: + @echo "Getting jar files into target ..." + @mkdir -p target + @cp ../../target/kafka-connect-hana-*.jar target + @cat driver-jars.txt | xargs -I '{}' mvn -q dependency:get -Dartifact='{}' -Dtransitive=false -Ddest=target diff --git a/examples/persons3/README.md b/examples/persons3/README.md index 9f748fd..12d2c07 100644 --- a/examples/persons3/README.md +++ b/examples/persons3/README.md @@ -6,7 +6,6 @@ This example is similar to example [persons2](../persons2) but example persons3 - This project is built (or its jar file is available) - Local Kafka installation -- Access to Apicurio schema registry - Access to HANA - Understanding of example [persons2](../persons2) @@ -81,35 +80,34 @@ $ ###### Add Apicurio registry client libraries to the plugin directory -In order to use Apicurio registry, its client libraries must be placed in the connector's plugin directory. The required jar files are listed in the [apicurio-registry-jars.txt](./apicurio-registry-jars.txt) file. +In order to use Apicurio registry, its client libraries must be placed in the connector's plugin directory. The required jar files are downloaded by running the following command. -Run the following command to download those jar files into the target directory. ``` -$ mkdir target && cat apicurio-registry-jars.txt | xargs -I '{}' mvn dependency:get -Dartifact='{}' -Dtransitive=false -Ddest=target -[INFO] Scanning for projects... -[INFO] -... +$ make get_libs +Getting jar files into target ... +$ +$ ls target +apicurio-registry-client-1.2.3.Final.jar jersey-common-2.29.1.jar +apicurio-registry-common-1.2.3.Final.jar jersey-hk2-2.29.1.jar +apicurio-registry-utils-converter-1.2.3.Final.jar jersey-media-jaxb-2.29.1.jar +apicurio-registry-utils-serde-1.2.3.Final.jar jersey-media-json-binding-2.29.1.jar +avro-1.9.2.jar jersey-mp-config-2.29.1.jar +cdi-api-2.0.jar jersey-mp-rest-client-2.29.1.jar +geronimo-config-impl-1.2.2.jar jersey-server-2.29.1.jar +jakarta.json-1.1.5.jar kafka-connect-hana-1.0-SNAPSHOT.jar +jakarta.json-api-1.1.5.jar microprofile-config-api-1.4.jar +jakarta.json.bind-api-1.0.2.jar microprofile-rest-client-api-1.4.0.jar +javax.interceptor-api-1.2.jar ngdbc-2.5.49.jar +jersey-cdi1x-2.29.1.jar yasson-1.0.3.jar +jersey-client-2.29.1.jar +$ ``` We copy the downloaded jar files into the connector's plugin directory. ``` -$ ls target -apicurio-registry-client-1.2.3.Final.jar jersey-client-2.29.1.jar -apicurio-registry-common-1.2.3.Final.jar jersey-common-2.29.1.jar -apicurio-registry-utils-converter-1.2.3.Final.jar jersey-hk2-2.29.1.jar -apicurio-registry-utils-serde-1.2.3.Final.jar jersey-media-jaxb-2.29.1.jar -avro-1.9.2.jar jersey-media-json-binding-2.29.1.jar -cdi-api-2.0.jar jersey-mp-config-2.29.1.jar -geronimo-config-impl-1.2.2.jar jersey-mp-rest-client-2.29.1.jar -jakarta.json-1.1.5.jar jersey-server-2.29.1.jar -jakarta.json-api-1.1.5.jar microprofile-config-api-1.4.jar -jakarta.json.bind-api-1.0.2.jar microprofile-rest-client-api-1.4.0.jar -javax.interceptor-api-1.2.jar yasson-1.0.3.jar -jersey-cdi1x-2.29.1.jar -$ -$ cp *.jar $KAFKA_HOME/plugins/kafka-connect-hana +$ cp target/*.jar $KAFKA_HOME/plugins/kafka-connect-hana $ ``` diff --git a/examples/persons3/apicurio-registry-jars.txt b/examples/persons3/driver-jars.txt similarity index 96% rename from examples/persons3/apicurio-registry-jars.txt rename to examples/persons3/driver-jars.txt index ded38fe..1f854a1 100644 --- a/examples/persons3/apicurio-registry-jars.txt +++ b/examples/persons3/driver-jars.txt @@ -1,3 +1,4 @@ +com.sap.cloud.db.jdbc:ngdbc:2.5.49 io.apicurio:apicurio-registry-client:1.2.3.Final io.apicurio:apicurio-registry-common:1.2.3.Final io.apicurio:apicurio-registry-utils-converter:1.2.3.Final diff --git a/examples/persons4/README.md b/examples/persons4/README.md index 4e5730b..2387238 100644 --- a/examples/persons4/README.md +++ b/examples/persons4/README.md @@ -6,7 +6,6 @@ This example is similar to example [persons3(../persons3) but example persons4 u - This project is built (or its jar file is available) - Local Kafka installation -- Access to Apicurio schema registry - Access to HANA - Understanding of example [persons2](../persons2) diff --git a/examples/persons4ds/Dockerfile b/examples/persons4ds/Dockerfile new file mode 100644 index 0000000..6367a1f --- /dev/null +++ b/examples/persons4ds/Dockerfile @@ -0,0 +1,7 @@ +FROM strimzi/kafka:0.19.0-kafka-2.4.1 +USER root:root + +RUN mkdir -p /opt/kafka/plugins/kafka-connect-hana +COPY ./target/ /opt/kafka/plugins/kafka-connect-hana/ + +USER 1001 diff --git a/examples/persons4ds/Makefile b/examples/persons4ds/Makefile new file mode 100644 index 0000000..99d13b4 --- /dev/null +++ b/examples/persons4ds/Makefile @@ -0,0 +1,19 @@ +# +# +# make get_libs - build the go binary +# make docker_build - build the docker image +# + +DOCKER_TAG ?= latest + +.PHONY: get_libs +get_libs: + @echo "Getting jar files into target ..." + @mkdir -p target + @cp ../../target/kafka-connect-hana-*.jar target + @cat driver-jars.txt | xargs -I '{}' mvn -q dependency:get -Dartifact='{}' -Dtransitive=false -Ddest=target + +.PHONY: docker_build +docker_build: + @echo "Building docker image ..." + docker build . -t strimzi-connector-hana-rega diff --git a/examples/persons4ds/README.md b/examples/persons4ds/README.md new file mode 100644 index 0000000..75d81ad --- /dev/null +++ b/examples/persons4ds/README.md @@ -0,0 +1,189 @@ +### Example persons4ds: kafka-hana-connect using strimzi kafka images with docker-compose + +This example is a distributed version of example persons4 using strimizi kafka docker images. + +#### Prerequisites + +- This project is built (or its jar file is available) +- Access to HANA +- Docker, Docker-Compose + +#### Running + +This description assumes Docker and Docker-Compose are available on local machine. + +##### Steps 1: Build Docker image for kafka-connect-hana + +First, run `make get_libs` to place the required jar files into directory `target`. + +``` +$ make get_libs +Getting jar files into target ... +$ +$ ls target +apicurio-registry-client-1.2.3.Final.jar jersey-common-2.29.1.jar +apicurio-registry-common-1.2.3.Final.jar jersey-hk2-2.29.1.jar +apicurio-registry-utils-converter-1.2.3.Final.jar jersey-media-jaxb-2.29.1.jar +apicurio-registry-utils-serde-1.2.3.Final.jar jersey-media-json-binding-2.29.1.jar +avro-1.9.2.jar jersey-mp-config-2.29.1.jar +cdi-api-2.0.jar jersey-mp-rest-client-2.29.1.jar +geronimo-config-impl-1.2.2.jar jersey-server-2.29.1.jar +jakarta.json-1.1.5.jar kafka-connect-hana-1.0-SNAPSHOT.jar +jakarta.json-api-1.1.5.jar microprofile-config-api-1.4.jar +jakarta.json.bind-api-1.0.2.jar microprofile-rest-client-api-1.4.0.jar +javax.interceptor-api-1.2.jar ngdbc-2.5.49.jar +jersey-cdi1x-2.29.1.jar yasson-1.0.3.jar +jersey-client-2.29.1.jar +$ +``` + +Next, run `make build_docker` to build the Docker image using strimzi's Kafka image and add the jar files to its plugins directory. + +``` +$ make docker_build +Building docker image ... +docker build . -t strimzi-connector-hana-min +Sending build context to Docker daemon 1.423MB +Step 1/5 : FROM strimzi/kafka:0.19.0-kafka-2.4.1 + ---> a0be3ed644fc +Step 2/5 : USER root:root + ---> Using cache + ---> e651c111cfd0 +Step 3/5 : RUN mkdir -p /opt/kafka/plugins/kafka-connect-hana + ---> Using cache + ---> b5e5ec331af9 +Step 4/5 : COPY ./target/ /opt/kafka/plugins/kafka-connect-hana/ + ---> Using cache + ---> d04b23041483 +Step 5/5 : USER 1001 + ---> Using cache + ---> 44fb0e9e5347 +Successfully built 44fb0e9e5347 +Successfully tagged strimzi-connector-hana-min:latest +$ +``` + +##### Step 2: Prepare the connector configuration files + +We use connect-distributed.properties stored in directory custom-config for configuring Kafka-Connect. For configuring connectors, prepare the json version of the connector configuration files connect-hana-source-1.json and connect-hana-sink-1.json. + +##### Step 3: Prepare the source table (Follow Step 4 of example persons1) + +##### Step 4: Starting Zookeeper, Kafka, Kafka-Coonnect + +The docker-compose.yaml file defines zookeeper, kafka, and connect services. It is noted that Kafka broker uses its advertised host set to `host.docker.internal:9092` assumeing this host name is resolvable from the containers and at the host. This allows Kafka broker to be accessed from the container of Kafka-Connect and from the host for inspection. + +Run `docker-compose up` to start the containers. + +``` +$ docker-compose up +Creating network "persons1ds_default" with the default driver +Creating persons1ds_zookeeper_1 ... done +Creating persons1ds_kafka_1 ... done +Creating persons1ds_connect_1 ... done +Attaching to persons1ds_zookeeper_1, persons1ds_kafka_1, persons1ds_connect_1 +zookeeper_1 | [2020-09-09 21:59:02,318] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) +... +``` + +After starting the Docker containers using docker-compose, we can verify whether Kafka Connect is running using curl. + +``` +$ curl -i http://localhost:8083/ +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:00:11 GMT +Content-Type: application/json +Content-Length: 91 +Server: Jetty(9.4.20.v20190813) + +{"version":"2.4.1","commit":"c57222ae8cd7866b","kafka_cluster_id":"wdrDgSAFSbKpWGYm-q0PuQ"} +$ +$ curl -i http://localhost:8083/connectors +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:00:39 GMT +Content-Type: application/json +Content-Length: 2 +Server: Jetty(9.4.20.v20190813) + +[] +$ +``` + +The above result shows that Kafka Connect using Kafka 2.4.1 is running and there is no connector deployed. + +We prepare for the connector json files using the json files `connect-hana-source-1.json` and `connect-hana-sink-1.json` which are the json representation of the property files created for example persons1. + +Finally, we deploy the connectors by posting the connector configuration json files to the Kafka Connect's API. Assuming, these json files are already prepared in step 3, use curl to post these files. + +``` +$ curl -i -X POST -H 'content-type:application/json' -d @connect-hana-source-4.json http://localhost:8083/connectors +HTTP/1.1 201 Created +Date: Wed, 09 Sep 2020 22:10:34 GMT +Location: http://localhost:8083/connectors/test-topic-4-source +Content-Type: application/json +Content-Length: 399 +Server: Jetty(9.4.20.v20190813) + +{"name":"test-topic-4-source","config":{"connector.class":"com.sap.kafka.connect.source.hana.HANASourceConnector","tasks.max":"1","topics":"test_topic_4","connection.url":"jdbc:sap://... +$ +$ curl -i -X POST -H 'content-type:application/json' -d @connect-hana-sink-4.json http://localhost:8083/connectors +HTTP/1.1 201 Created +Date: Wed, 09 Sep 2020 22:11:22 GMT +Location: http://localhost:8083/connectors/test-topic-4-sink +Content-Type: application/json +Content-Length: 414 +Server: Jetty(9.4.20.v20190813) + +{"name":"test-topic-4-sink","config":{"connector.class":"com.sap.kafka.connect.sink.hana.HANASinkConnector","tasks.max":"1","topics":"test_topic_4","connection.url":"jdbc:sap://... +$ +$ curl -i http://localhost:8083/connectors +HTTP/1.1 200 OK +Date: Wed, 09 Sep 2020 22:11:54 GMT +Content-Type: application/json +Content-Length: 43 +Server: Jetty(9.4.20.v20190813) + +["test-topic-4-source","test-topic-4-sink"] +``` + +The above result shows that the connectors are deployed. + + +##### Step 5: Verifying the result (Follow Step 6 of example persons4) + +In addition to inspecting Kafka topic and HANA table, you can verify the registered schema at the schema registry using the following command. + +``` +$ curl -i http://localhost:8080/api/artifacts +HTTP/1.1 200 OK +Date: Thu, 10 Sep 2020 12:17:36 GMT +Expires: Wed, 09 Sep 2020 12:17:36 GMT +Pragma: no-cache +Cache-control: no-cache, no-store, must-revalidate +Content-Type: application/json +Content-Length: 43 + +["test_topic_4-value"] +$ +``` + +It is noted that this scenario builds the Docker image with the apicurio schema registry usage and runs Kafka Connect in the distributed mode. Additional connectors can be deployed to this Kafka Connect instance which use the same distributed-connect.properties configuration. + + +##### Step 6: Shut down + +Use `docker-compose down` to shutdown the containers. + +``` +$ docker-compose down +Stopping persons4ds_registry_1 ... done +Stopping persons4ds_connect_1 ... done +Stopping persons4ds_kafka_1 ... done +Stopping persons4ds_zookeeper_1 ... done +Removing persons4ds_registry_1 ... done +Removing persons4ds_connect_1 ... done +Removing persons4ds_kafka_1 ... done +Removing persons4ds_zookeeper_1 ... done +Removing network persons4ds_default +$ +``` diff --git a/examples/persons4ds/connect-hana-sink-4.json b/examples/persons4ds/connect-hana-sink-4.json new file mode 100644 index 0000000..a09b3ac --- /dev/null +++ b/examples/persons4ds/connect-hana-sink-4.json @@ -0,0 +1,14 @@ +{ + "name": "test-topic-4-sink", + "config": { + "connector.class": "com.sap.kafka.connect.source.hana.HANASourceConnector", + "tasks.max": "1", + "topics": "test_topic_4", + "connection.url": "jdbc:sap:///", + "connection.user": "", + "connection.password": "", + "auto.create": "true", + "test_topic_4.table.name": "\"\".\"PERSONS4_RES\"" + } +} + diff --git a/examples/persons4ds/connect-hana-source-4.json b/examples/persons4ds/connect-hana-source-4.json new file mode 100644 index 0000000..5ab36d0 --- /dev/null +++ b/examples/persons4ds/connect-hana-source-4.json @@ -0,0 +1,12 @@ +{ + "name": "test-topic-4-source", + "config": { + "connector.class": "com.sap.kafka.connect.source.hana.HANASourceConnector", + "tasks.max": "1", + "topics": "test_topic_4", + "connection.url": "jdbc:sap:///", + "connection.user": "", + "connection.password": "", + "test_topic_4.table.name": "\"\".\"PERSONS4\"" + } +} diff --git a/examples/persons4ds/custom-config/connect-distributed.properties b/examples/persons4ds/custom-config/connect-distributed.properties new file mode 100644 index 0000000..f60f137 --- /dev/null +++ b/examples/persons4ds/custom-config/connect-distributed.properties @@ -0,0 +1,92 @@ +## +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## + +# This file contains some of the configurations for the Kafka Connect distributed worker. This file is intended +# to be used with the examples, and some settings may differ from those used in a production system, especially +# the `bootstrap.servers` and those specifying replication factors. + +# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. +bootstrap.servers=kafka:9092 + +# unique name for the cluster, used in forming the Connect cluster group. Note that this must not conflict with consumer group IDs +group.id=connect-cluster + +# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will +# need to configure these based on the format they want their data in when loaded from or stored into Kafka +key.converter=org.apache.kafka.connect.json.JsonConverter +value.converter=io.apicurio.registry.utils.converter.AvroConverter +# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply +# it to +key.converter.schemas.enable=true +value.converter.schemas.enable=true + +# use apicurio registry +value.converter.apicurio.registry.url=http://registry:8080/api +value.converter.apicurio.registry.converter.serializer=io.apicurio.registry.utils.serde.AvroKafkaSerializer +value.converter.apicurio.registry.converter.deserializer=io.apicurio.registry.utils.serde.AvroKafkaDeserializer +value.converter.apicurio.registry.global-id=io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy + +# Topic to use for storing offsets. This topic should have many partitions and be replicated and compacted. +# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create +# the topic before starting Kafka Connect if a specific topic configuration is needed. +# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value. +# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able +# to run this example on a single-broker cluster and so here we instead set the replication factor to 1. +offset.storage.topic=connect-offsets +offset.storage.replication.factor=1 +#offset.storage.partitions=25 + +# Topic to use for storing connector and task configurations; note that this should be a single partition, highly replicated, +# and compacted topic. Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create +# the topic before starting Kafka Connect if a specific topic configuration is needed. +# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value. +# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able +# to run this example on a single-broker cluster and so here we instead set the replication factor to 1. +config.storage.topic=connect-configs +config.storage.replication.factor=1 + +# Topic to use for storing statuses. This topic can have multiple partitions and should be replicated and compacted. +# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create +# the topic before starting Kafka Connect if a specific topic configuration is needed. +# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value. +# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able +# to run this example on a single-broker cluster and so here we instead set the replication factor to 1. +status.storage.topic=connect-status +status.storage.replication.factor=1 +#status.storage.partitions=5 + +# Flush much faster than normal, which is useful for testing/debugging +offset.flush.interval.ms=10000 + +# These are provided to inform the user about the presence of the REST host and port configs +# Hostname & Port for the REST API to listen on. If this is set, it will bind to the interface used to listen to requests. +#rest.host.name= +rest.port=8083 + +# The Hostname & Port that will be given out to other workers to connect to i.e. URLs that are routable from other servers. +#rest.advertised.host.name= +#rest.advertised.port= + +# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins +# (connectors, converters, transformations). The list should consist of top level directories that include +# any combination of: +# a) directories immediately containing jars with plugins and their dependencies +# b) uber-jars with plugins and their dependencies +# c) directories immediately containing the package directory structure of classes of plugins and their dependencies +# Examples: +# plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors, +plugin.path=/opt/kafka/plugins diff --git a/examples/persons4ds/docker-compose.yaml b/examples/persons4ds/docker-compose.yaml new file mode 100644 index 0000000..4195463 --- /dev/null +++ b/examples/persons4ds/docker-compose.yaml @@ -0,0 +1,55 @@ +version: '2' + +services: + + zookeeper: + image: strimzi/kafka:0.19.0-kafka-2.4.1 + command: [ + "sh", "-c", + "bin/zookeeper-server-start.sh config/zookeeper.properties" + ] + ports: + - "2181:2181" + environment: + LOG_DIR: /tmp/logs + + kafka: + image: strimzi/kafka:0.19.0-kafka-2.4.1 + command: [ + "sh", "-c", + "bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}" + ] + depends_on: + - zookeeper + ports: + - "9092:9092" + environment: + LOG_DIR: "/tmp/logs" + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://host.docker.internal:9092 + KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 + KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 + + registry: + image: apicurio/apicurio-registry-mem:1.2.3.Final + depends_on: + - kafka + ports: + - "8080:8080" + + connect: + image: strimzi-connector-hana-rega:latest + volumes: + - ./custom-config:/opt/kafka/custom-config + command: [ + "sh", "-c", + "bin/connect-distributed.sh custom-config/connect-distributed.properties" + ] + depends_on: + - kafka + ports: + - "8083:8083" + environment: + LOG_DIR: "/tmp/logs" + +volumes: + custom-config: \ No newline at end of file diff --git a/examples/persons4ds/driver-jars.txt b/examples/persons4ds/driver-jars.txt new file mode 100644 index 0000000..1f854a1 --- /dev/null +++ b/examples/persons4ds/driver-jars.txt @@ -0,0 +1,24 @@ +com.sap.cloud.db.jdbc:ngdbc:2.5.49 +io.apicurio:apicurio-registry-client:1.2.3.Final +io.apicurio:apicurio-registry-common:1.2.3.Final +io.apicurio:apicurio-registry-utils-converter:1.2.3.Final +io.apicurio:apicurio-registry-utils-serde:1.2.3.Final +org.eclipse:yasson:1.0.3 +javax.interceptor:javax.interceptor-api:1.2 +jakarta.json.bind:jakarta.json.bind-api:1.0.2 +org.glassfish.jersey.media:jersey-media-json-binding:2.29.1 +jakarta.json:jakarta.json-api:1.1.5 +org.glassfish:jakarta.json:1.1.5 +org.apache.avro:avro:1.9.2 +javax.enterprise:cdi-api:2.0 +org.apache.geronimo.config:geronimo-config-impl:1.2.2 +org.glassfish.jersey.ext.cdi:jersey-cdi1x:2.29.1 +org.glassfish.jersey.core:jersey-client:2.29.1 +org.glassfish.jersey.core:jersey-common:2.29.1 +org.glassfish.jersey.inject:jersey-hk2:2.29.1 +org.glassfish.jersey.media:jersey-media-jaxb:2.29.1 +org.glassfish.jersey.ext.microprofile:jersey-mp-config:2.29.1 +org.glassfish.jersey.ext.microprofile:jersey-mp-rest-client:2.29.1 +org.glassfish.jersey.core:jersey-server:2.29.1 +org.eclipse.microprofile.config:microprofile-config-api:1.4 +org.eclipse.microprofile.rest.client:microprofile-rest-client-api:1.4.0 diff --git a/examples/persons6/README.md b/examples/persons6/README.md index 9130954..17db3f4 100644 --- a/examples/persons6/README.md +++ b/examples/persons6/README.md @@ -42,21 +42,20 @@ In order to use Apicurio registry, its client libraries must be placed in the co Run the following command to download those jar files into the target directory. ``` -$ mkdir target && cat confluent-registry-jars.txt | xargs -I '{}' mvn dependency:get -Dartifact='{}' -Dtransitive=false -DremoteRepositories=confluent::::https://packages.confluent.io/maven -Ddest=target -[INFO] Scanning for projects... -[INFO] -... +$ make get_libs +Getting jar files into target ... +$ ls target +avro-1.9.2.jar kafka-connect-avro-converter-5.4.2.jar +common-config-5.4.2.jar kafka-connect-hana-1.0-SNAPSHOT.jar +common-utils-5.4.2.jar kafka-schema-registry-client-5.4.2.jar +kafka-avro-serializer-5.4.2.jar ngdbc-2.5.49.jar +$ ``` We copy the downloaded jar files into the connector's plugin directory. ``` -$ ls target -avro-1.9.2.jar kafka-avro-serializer-5.4.2.jar -common-config-5.4.2.jar kafka-connect-avro-converter-5.4.2.jar -common-utils-5.4.2.jar kafka-schema-registry-client-5.4.2.jar -$ -$ cp *.jar $KAFKA_HOME/plugins/kafka-connect-hana +$ cp target/*.jar $KAFKA_HOME/plugins/kafka-connect-hana $ $ cd $KAFKA_HOME $ ls plugins/kafka-connect-hana diff --git a/examples/persons6/confluent-registry-jars.txt b/examples/persons6/confluent-registry-jars.txt deleted file mode 100644 index df776f3..0000000 --- a/examples/persons6/confluent-registry-jars.txt +++ /dev/null @@ -1,7 +0,0 @@ -io.confluent:kafka-connect-avro-converter:5.4.2 -io.confluent:common-config:5.4.2 -io.confluent:common-utils:5.4.2 -io.confluent:kafka-schema-registry-client:5.4.2 -io.confluent:kafka-avro-serializer:5.4.2 -org.apache.avro:avro:1.9.2 -